{ "cells": [ { "cell_type": "markdown", "id": "438ec247", "metadata": {}, "source": [ "# SampleData Reference Sheet" ] }, { "cell_type": "markdown", "id": "f0ec5da0", "metadata": {}, "source": [ "## Notation" ] }, { "cell_type": "markdown", "id": "3348824b", "metadata": {}, "source": [ "In the code lines constituting this reference sheet, the following notation conventions are used in the SampleData method arguments:\n", "\n", "* `node_name`, `group_name`, `data_item_name` are the Name, Indexname, Path or Alias of respectively a *Data Array*, *Group*, or either of the two previous data items.\n", "* `parent_name` is the the Name, Indexname, Path or Alias of a group that is the parent of the \n", "* `data` is a *SampleData* class instance synchronized with the dataset files `my_dataset.hdf5` and `my_dataset.xdmf`." ] }, { "cell_type": "markdown", "id": "446c2721", "metadata": {}, "source": [ "### SampleData Naming system" ] }, { "cell_type": "markdown", "id": "27fe8a00", "metadata": {}, "source": [ "Interacting with data item in a *SampleData* dataset require to provide their name to the various class method. 4 possible types of names can be provided for each data item: \n", "\n", "1. the **Path** of the data item in the HDF5 file. \n", "2. the **Name** of the data item.\n", "3. the **Indexname** of the data item\n", "4. the **Alias** or aliases of the data item" ] }, { "cell_type": "markdown", "id": "8007ef13", "metadata": {}, "source": [ "## Creating, Opening datasets, and Exploring their content" ] }, { "cell_type": "markdown", "id": "2cd3ba66", "metadata": {}, "source": [ "### Dataset creation/opening" ] }, { "cell_type": "markdown", "id": "b5afbac3", "metadata": {}, "source": [ "Import *SampleData* class:" ] }, { "cell_type": "code", "execution_count": null, "id": "56314ca6", "metadata": {}, "outputs": [], "source": [ "from pymicro.core.samples import SampleData as SD" ] }, { "cell_type": "markdown", "id": "8665643f", "metadata": {}, "source": [ "Create/Open a *SampleData* dataset, and activate verbose mode:" ] }, { "cell_type": "code", "execution_count": null, "id": "7c871ac6", "metadata": {}, "outputs": [], "source": [ "# CREATE dataset: the file `filename` must not exist. Verbose mode OFF\n", "data = SD(filename='my_first_dataset', verbose=False) \n", "# OPEN dataset: the file `filename` must exist. Verbose mode ON\n", "data = SD(filename='my_first_dataset', verbose=True) " ] }, { "cell_type": "markdown", "id": "71bfb1f7", "metadata": {}, "source": [ "Copy dataset and get class instance synchronized with new dataset:" ] }, { "cell_type": "code", "execution_count": null, "id": "ab7fbb92", "metadata": {}, "outputs": [], "source": [ "data = SD.copy_sample(src_sample_file='source_dataset', dst_sample_file='destination_dataset', get_object=True)" ] }, { "cell_type": "markdown", "id": "583a8f9c", "metadata": {}, "source": [ "Create dataset, overwrite existing file, and automatic removal of dataset files at class instance destruction (autodelete option):" ] }, { "cell_type": "code", "execution_count": null, "id": "f7961bbf", "metadata": {}, "outputs": [], "source": [ "# Create new dataset and overwrite already existing dataset files \n", "data = SD(filename='my_first_dataset', verbose=True, overwrite_hdf5=True)\n", "# Create new dataset with autodelete option ON\n", "data = SD(filename='my_first_dataset', verbose=True, autodelete=True)\n", "# Set autodelete option on\n", "data.autodelete = True" ] }, { "cell_type": "markdown", "id": "e1f15677", "metadata": {}, "source": [ "### Getting information on datasets" ] }, { "cell_type": "markdown", "id": "6cb40148", "metadata": {}, "source": [ "Print informations on global content of the dataset:" ] }, { "cell_type": "code", "execution_count": null, "id": "1eaac75d", "metadata": {}, "outputs": [], "source": [ "# Print dataset index\n", "data.print_index() # --> no option = local root '/' and max depth 3\n", "data.print_index(max_depth=2, local_root='local_root_name') # --> with specified local root and depth\n", "\n", "# Print dataset content (list all groups, nodes with detailed information)\n", "data.print_dataset_content() # detailed output, printed in standard output \n", "data.print_dataset_content(short=True, to_file='dataset_information.txt') # short output, written in text file\n", "\n", "# Print both index and dataset content in short version --> class string representation\n", "print(data)\n", "\n", "# Print only grid groups informations\n", "data.print_grids_info()\n", "\n", "# Print content of XDMF tree\n", "data.print_xdmf()\n", "\n", "# Get the memory disk size of the HDF5 dataset\n", "size, unit = data.get_file_disk_size(convert=True, print_flag=False) \n", "# value not printed, returned in method output and converted to most readable memory unit\n" ] }, { "cell_type": "markdown", "id": "721d5ec8", "metadata": {}, "source": [ "Command line tools to get information on dataset" ] }, { "cell_type": "code", "execution_count": null, "id": "3b2b13af", "metadata": {}, "outputs": [], "source": [ "# recursive (-r) and detailed (-d) output of h5ls --> also print the content of the data arrays\n", "!h5ls -rd ../data/test_sampledata_ref.h5\n", "# h5dump\n", "!ptdump -d ../data/test_sampledata_ref.h5\n", "# detailed (-d) and verbose (-v) output of ptdump\n", "!ptdump -dv ../data/test_sampledata_ref.h5" ] }, { "cell_type": "code", "execution_count": null, "id": "9838e182", "metadata": {}, "outputs": [], "source": [ "# Print information on data items content\n", "data.print_node_info(nodename='node_name') # detailed information on a data item (group or array node)\n", "data.print_group_content('group_name') # print information on group childrens\n", "data.print_group_content('group_name', recursive=True) # print information on group childrens recursively \n", "\n", "# get data item disk size\n", "size, unit = data.get_node_disk_size(nodename='node_name', print_flag=False, convert=False)" ] }, { "cell_type": "markdown", "id": "f9b3132b", "metadata": {}, "source": [ "### Dataset interactive visualization" ] }, { "cell_type": "code", "execution_count": null, "id": "c6a865de", "metadata": {}, "outputs": [], "source": [ "# Visualize dataset organization and content with Vitables\n", "data.pause_for_visualization(Vitables=True, Vitables_path='Path_to_Vitables_executable')\n", "# Visualize spatially organized data with Paraview\n", "data.pause_for_visualization(Paraview=True, Paraview_path='Path_to_Paraview_executable')" ] }, { "cell_type": "markdown", "id": "5752f4d0", "metadata": {}, "source": [ "## Basic data items: creating and getting them" ] }, { "cell_type": "markdown", "id": "59d1a0a9", "metadata": {}, "source": [ "### Generic methods to get data items" ] }, { "cell_type": "code", "execution_count": null, "id": "7b56f7ed", "metadata": {}, "outputs": [], "source": [ "# Dictionary like access\n", "data['data_item_name']\n", "# Attribute like access\n", "data.data_item_name\n", "# generic getter method \n", "data.get_node('data_item_name') # --> returns a Pytables Group or Node object\n", "data.get_node('data_item_name', as_numpy=True) # --> for array data items, returns a numpy array" ] }, { "cell_type": "markdown", "id": "bbd64046", "metadata": {}, "source": [ "### Group data items" ] }, { "cell_type": "markdown", "id": "ee855d74", "metadata": {}, "source": [ "`group_name` and `group_indexnam` are the Name and Indexname of a *Group* data item. `parent_group` is the Name, Path, Indexname or Alias of a *Group* where `group_name` will be stored. " ] }, { "cell_type": "code", "execution_count": null, "id": "e854edc2", "metadata": {}, "outputs": [], "source": [ "# Create a group in a dataset with name `group_name`, stored in the group `parent_name`\n", "data.add_group(groupname='group_name', location='parent_name', indexname='group_indexname')\n", "# Create a group and overwrite pre-existing group with the same name + get the created Pytables Group object\n", "group = data.add_group(groupname='group_name', location='parent_name', indexname='group_indexname', replace=True)" ] }, { "cell_type": "markdown", "id": "b3ea95a0", "metadata": {}, "source": [ "### Data item attributes (metadata)" ] }, { "cell_type": "code", "execution_count": null, "id": "d3aded0b", "metadata": {}, "outputs": [], "source": [ "# Add attributes from a python dictionary (metadata_dictionary)\n", "data.add_attributes(metadata_dictionary, nodename='node_name')\n", "\n", "# get data item attributes (metadata)\n", "data.print_node_attributes(nodename='node_name') # print all attributes of note\n", "attribute_value = data.get_attribute(attrname='attribute_name', nodename='node_name') # get value of one attribute\n", "mesh_attrs = data.get_dic_from_attributes(nodename='node_name') # get all attributes as a dictionary\n", "\n", "# set and get specific `description` attribute for node `node_name`\n", "data.set_description(description=\"Write your description text here.\", node='node_name')\n", "data.get_description('node_name')" ] }, { "cell_type": "markdown", "id": "bd479b64", "metadata": {}, "source": [ "### Data arrays" ] }, { "cell_type": "code", "execution_count": null, "id": "f099ef02", "metadata": {}, "outputs": [], "source": [ "# add a numpy array `array` in data item `node_name`\n", "data.add_data_array(location='parent_name', name='node_name', indexname='array_indexname', array=array)\n", " # replace = True allows to overwrite preexisting field with same name\n", "data.add_data_array(location='parent_name', name='node_name', indexname='array_indexname', array=array, replace= True)\n", "\n", "# get data array from data item `node_name`\n", "array_node = data.get_node('node_name') # --> returns a Pytables Node object\n", "array = data.get_node('node_name', as_numpy=True) # --> returns a Numpy array\n", "array = data.['node_name'] # --> returns a Numpy array\n", "array = data.node_name # --> returns a Numpy array " ] }, { "cell_type": "markdown", "id": "492a02a4", "metadata": {}, "source": [ "### String arrays" ] }, { "cell_type": "code", "execution_count": null, "id": "e68a915f", "metadata": {}, "outputs": [], "source": [ "# Add list of strings `List` as a string array `node_name` \n", "data.add_string_array(name='node_name', location='parent_name', indexname='Sarray_indexname', data=List)\n", "\n", "# get and decode binary strings stored in a String array\n", "sarray = data['Sarray_indexname']\n", "for string in sarray:\n", " print(string.decode('utf-8'), end=' ') # prints each string of the String array" ] }, { "cell_type": "markdown", "id": "c49034b8", "metadata": {}, "source": [ "### Structured arrays" ] }, { "cell_type": "code", "execution_count": null, "id": "cb2c6af6", "metadata": {}, "outputs": [], "source": [ "# Add structured array from Numpy structured array `structured_array` with Numpy.dtype `table_type`\n", "data.add_table(name='node_name', location='parent_name', indexname='table_indexname', description=table_type,\n", " data=structured_array)\n", "# Add lines to a structured array node from Numpy array `structured_array` (same dtype as the table)\n", "data.append_table(name='table_indexname', data=structured_array)\n", "\n", "# Add columns to a structured array node from a Numpy array `structured_array` with Numpy.dtype ``\n", "data.add_tablecols(tablename='table_indexname', description=cols_dtype, data=structured_array)\n", "\n", "# Get structured array just like Data arrays" ] }, { "cell_type": "markdown", "id": "8da12df5", "metadata": {}, "source": [ "### Remove data items" ] }, { "cell_type": "code", "execution_count": null, "id": "263758df", "metadata": {}, "outputs": [], "source": [ "data.remove_node('node_name') # removes a group without childrens or a data array item\n", "data.remove_node('group_name', recursive=True) # remove a Group data item and all its childrens recursively\n", "# remove one or a list of attributes (metadata) from a node\n", "data.remove_attribute(attrname='attribute_name', nodename='node_name')\n", "data.remove_attributes(attr_list=['list','of','attributes','to','remove','from','node'], nodename='node_name')" ] }, { "cell_type": "markdown", "id": "0e57d8c9", "metadata": {}, "source": [ "## Image Groups and Image fields: creating and getting them" ] }, { "cell_type": "markdown", "id": "a98fd61b", "metadata": {}, "source": [ "### Creating Image groups from fields" ] }, { "cell_type": "code", "execution_count": null, "id": "936b809a", "metadata": {}, "outputs": [], "source": [ "# Create an Image Group from a Numpy array `field_array` interpreted as a pixel/voxel wise constant scalar field\n", "data.add_image_from_field(field_array=field_array, fieldname='node_name', imagename='group_name',\n", " indexname='image_indexname', location='parent_name', \n", " description=\"Write image group description here.\", origin=[0.,10.], spacing=[2.,2.])\n", "\n", "# Create an Image Group from a Numpy array `field_array` interpreted as a node value scalar field\n", "data.add_image_from_field(field_array=field_array, fieldname='node_name', imagename='group_name',\n", " indexname='image_indexname', location='parent_name', is_elemField=False,\n", " description=\"Write image group description here.\", origin=[0.,10.], spacing=[2.,2.])\n", "\n", "# Create an Image Group from a Numpy array `field_array` interpreted as a non scalar field\n", "data.add_image_from_field(field_array=field_array, fieldname='node_name', imagename='group_name',\n", " indexname='image_indexname', location='parent_name', is_scalar=False,\n", " description=\"Write image group description here.\", origin=[0.,10.], spacing=[2.,2.])\n", "\n", "# Set image position and dimensions\n", "data.set_voxel_size(image_group='image_indexname', voxel_size=np.array([4.,4.]))\n", "data.set_origin(image_group='image_indexname', origin=np.array([10.,0.]))" ] }, { "cell_type": "markdown", "id": "dcad5186", "metadata": {}, "source": [ "### Creating image groups from image objects" ] }, { "cell_type": "code", "execution_count": null, "id": "467639cf", "metadata": {}, "outputs": [], "source": [ "# import BasicTools image object `ConstantRectilinearMesh`\n", "from BasicTools.Containers.ConstantRectilinearMesh import ConstantRectilinearMesh\n", "# Initialize image object, image dimension, origin and pixel/voxel size\n", "image_object = ConstantRectilinearMesh(dim=3)\n", "image_object.SetDimensions((50,50,3))\n", "image_object.SetOrigin([0.,0.,0.])\n", "image_object.SetSpacing([1.,1.,1.]) # pixel/voxel size in each dimension\n", "# Create Image Group in dataset\n", "data.add_image(image_object, imagename='group_name', indexname='image_indexname', location='parent_name', \n", " description=\"\"\"Write image group description here.\"\"\")" ] }, { "cell_type": "markdown", "id": "0df24564", "metadata": {}, "source": [ "### Creating empty images" ] }, { "cell_type": "code", "execution_count": null, "id": "7166dcbb", "metadata": {}, "outputs": [], "source": [ "data.add_image(imagename='group_name', indexname='image_indexname', location='parent_name',\n", " description=\"\"\"Write image group description here.\"\"\")" ] }, { "cell_type": "markdown", "id": "e37e05d9", "metadata": {}, "source": [ "### Get image object from Image Group" ] }, { "cell_type": "code", "execution_count": null, "id": "1c2f0290", "metadata": {}, "outputs": [], "source": [ "# Get BasicTools image object from SampleData image group `group_name` including image group fields data arrays \n", "im_object = data.get_image('group_name', with_fields=True)" ] }, { "cell_type": "markdown", "id": "8918795e", "metadata": {}, "source": [ "### Creating and getting image Fields" ] }, { "cell_type": "code", "execution_count": null, "id": "8df2a84e", "metadata": {}, "outputs": [], "source": [ "# Creating a field for image group `group_name` from Numpy array `tensor_field`\n", "data.add_field(gridname='group_name', fieldname='node_name', location='parent_name', indexname='field_indexname',\n", " array=tensor_field, replace=True) # replace = True allows to overwrite preexisting field with same name\n", "\n", "# Getting image fields \n", "# --> field returned as Numpy array\n", "field = data.get_field('node_name') \n", "field = data.get_node('node_name', as_numpy=True)\n", "field = data['node_name']\n", "field = data.node_name\n", "# --> field returned as a Pytables Node object\n", "field = data.get_node('node_name')" ] }, { "cell_type": "markdown", "id": "19d06929", "metadata": {}, "source": [ "### Creating a field time serie" ] }, { "cell_type": "code", "execution_count": null, "id": "5b34e00b", "metadata": {}, "outputs": [], "source": [ "instants = [1.,10., 100.]\n", "# Add three temporal values for the field `node_name` in image group `group_name` for 3 different time values given in\n", "# `instants` array. Field values are stored in *Numpy* arrays temporal_field_0, temporal_field_1, temporal_field_2\n", "data.add_field(gridname='group_name', fieldname='node_name', location='parent_name', indexname='Field',\n", " array=temporal_field_0, time=instants[0])\n", "# instant 1\n", "data.add_field(gridname='group_name', fieldname='node_name', location='parent_name', indexname='Field',\n", " array=temporal_field_1, time=instants[1])\n", "# instant 2\n", "data.add_field(gridname='group_name', fieldname='node_name', location='parent_name', indexname='Field',\n", " array=temporal_field_2, time=instants[2])" ] }, { "cell_type": "markdown", "id": "11c9347c", "metadata": {}, "source": [ "## Mesh Groups and Mesh Fields: creating and getting them" ] }, { "cell_type": "markdown", "id": "9a201e2e", "metadata": {}, "source": [ "### Creating Mesh objects with BasicTools" ] }, { "cell_type": "code", "execution_count": null, "id": "b175b5f3", "metadata": {}, "outputs": [], "source": [ "# Import Basictools mesh creation tools\n", "import BasicTools.Containers.UnstructuredMeshCreationTools as UMCT\n", "# Create a Node and Connectivity (elements) array, then create a mesh:\n", "mesh = UMCT.CreateMeshOfTriangles(mesh_nodes, mesh_elements) # mesh of triangles\n", "mesh = UMCT.CreateMeshOf(mesh_nodes, mesh_elements, elemName='tet4') # mesh of tetrahedra\n", "# Create a mesh of a cube with tetrahedron elements\n", "mesh = UMCT.CreateCube(dimensions=[5,5,5],spacing=[2.,2.,2.],ofTetras=True) \n", "\n", "# adding node and element tags to the mesh\n", "mesh.nodesTags.CreateTag('nodetag_name', False).SetIds(nodetag_Id_list) # Node tag\n", "mesh.GetElementsOfType('tri3').GetTag('elemtag_name').SetIds(elemtag_Id_list) # Element tag ( of type `tri3`)\n", "\n", "# adding fields\n", "mesh.nodeFields['nodal_fieldname'] = nodal_field_array\n", "mesh.elemFields['element_fieldname'] = elem_field_array" ] }, { "cell_type": "markdown", "id": "6595a32b", "metadata": {}, "source": [ "### Creating a Mesh Group in a dataset" ] }, { "cell_type": "code", "execution_count": null, "id": "b615cadd", "metadata": {}, "outputs": [], "source": [ "# Creating Mesh Group from Mesh object\n", "# mesh is a Basictools mesh object. bin_fields_from_sets options allows to load node and element tags in mesh Group\n", "data.add_mesh(mesh_object=mesh, meshname='meshname', indexname='mesh_indexname', location='mesh_parent',\n", " bin_fields_from_sets=True)\n", "\n", "# Creating Mesh group from file\n", "data.add_mesh(file=meshfile_name, meshname='meshname', indexname='mesh_indexname', location='mesh_parent', \n", " bin_fields_from_sets=True)" ] }, { "cell_type": "markdown", "id": "a0544a14", "metadata": {}, "source": [ "### Creating and getting Mesh Fields" ] }, { "cell_type": "code", "execution_count": null, "id": "b627972b", "metadata": {}, "outputs": [], "source": [ "# creation of the mesh field\n", "data.add_field(gridname='meshname', fieldname='fieldname', array=field_data_array, indexname='field_indexname')\n", "# Creation of a field part of a time serie\n", "data.add_field(gridname='meshname', fieldname='fieldname', array=field_data_array, indexname='field_indexname',\n", " time)\n", "# Force element field to be defined on boundary elements if the mesh has same number of bulk and boundary elements\n", "data.add_field(gridname='meshname', fieldname='fieldname', array=field_data_array, indexname='field_indexname',\n", " time)\n", "\n", "# getting the inputed array --> no options\n", "field_data_array = data.get_field('fieldname')\n", "# getting the visualization array of an integration point field\n", "field_data_array = data.get_field('fieldname', get_visualisation_field=True)\n", "# getting the unpadded visualization array of an integration point field\n", "field_data_array = data.get_field('fieldname', unpad_field=False, get_visualisation_field=True)" ] }, { "cell_type": "markdown", "id": "7f8c0e10", "metadata": {}, "source": [ "### Getting Mesh objects" ] }, { "cell_type": "code", "execution_count": null, "id": "dfaa62f9", "metadata": {}, "outputs": [], "source": [ "# Get a Basictools mesh object with all content of Mesh group 'meshname' (fields, tags, nodes, elements)\n", "mesh = data.get_mesh('meshname')\n", "# Get a Basictools mesh object without fields (tags, nodes, elements) from Mesh group 'meshname'\n", "mesh = data.get_mesh('meshname', with_fields=False)\n", "# Get a Basictools mesh object without fields and tags (just nodes, elements) from Mesh group 'meshname'\n", "mesh = data.get_mesh('meshname', with_fields=False, with_tags=False)" ] }, { "cell_type": "markdown", "id": "8c551e7b", "metadata": {}, "source": [ "## Data Compression " ] }, { "cell_type": "code", "execution_count": null, "id": "59b76261", "metadata": {}, "outputs": [], "source": [ "# Set chunckshape and compression settings for one data item\n", "compression_options = {'complib':'zlib', 'complevel':1, 'shuffle':True}\n", "chunkshape = c_shap # tuple\n", "data.set_chunkshape_and_compression(nodename='nodename_to_compress', compression_options=compression_options,\n", " chunkshape=chunkshape)\n", "\n", "# Set chunckshape and compression settings forseveral nodes\n", "compression_options = {'complib':'zlib', 'complevel':1, 'shuffle':True}\n", "data.set_nodes_compression_chunkshape(node_list=['nodename_to_compress1', 'nodename_to_compress2',...],\n", " compression_options=compression_options, \n", " chunkshape=chunkshape)\n", "\n", "# Apply lossy compression\n", "compression_options = {'complib':'zlib', 'complevel':1, 'shuffle':True, 'least_significant_digit':2}\n", "data.set_chunkshape_and_compression(nodename='nodename_to_compress', compression_options=compression_options,\n", " chunkshape=chunkshape)\n", "\n", "# Apply lossy compression with normalization\n", "compression_options = {'complib':'zlib', 'complevel':1, 'shuffle':True, 'least_significant_digit':2,\n", " normalization='standard'}\n", "data.set_chunkshape_and_compression(nodename='nodename_to_compress', compression_options=compression_options,\n", " chunkshape=chunkshape)\n", "\n", "\n", "# Apply lossy compression with per-component normalization\n", "compression_options = {'complib':'zlib', 'complevel':1, 'shuffle':True, 'least_significant_digit':2,\n", " normalization='standard-per-component'}\n", "data.set_chunkshape_and_compression(nodename='nodename_to_compress', compression_options=compression_options,\n", " chunkshape=chunkshape)\n", "\n", "# Create an array with predefined chunkshape and compression settings \n", "data.add_data_array(name='arrayname', indexname='array_indexname', location='parent_name', array=array,\n", " chunkshape=chunkshape, compression_options=compression_options)" ] } ], "metadata": { "kernelspec": { "display_name": "env_dev", "language": "python", "name": "env_dev" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.17" }, "nbsphinx": { "execute": "never" } }, "nbformat": 4, "nbformat_minor": 5 }