{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "e3676a78",
"metadata": {},
"source": [
"# Pymicro's Datasets"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6b50bf2e",
"metadata": {},
"source": [
"This first tutorial will introduce you to the creation and deletion of Pymicro's datasets. "
]
},
{
"cell_type": "markdown",
"id": "c9732f16",
"metadata": {},
"source": [
"## I - Create and Open datasets with the SampleData class"
]
},
{
"cell_type": "markdown",
"id": "a6171f3c",
"metadata": {},
"source": [
"In this first section, we will see how to create *SampleData* datasets, or open pre-existing ones. These two operations are performed by instantiating a SampleData class object. \n",
"\n",
"Before that, you will need to import the `SampleData` class. We will import it with the alias name `SD`, by executing:"
]
},
{
"cell_type": "markdown",
"id": "3086b5e4",
"metadata": {},
"source": [
"### Import SampleData and get help"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5af9e02b",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"from pymicro.core.samples import SampleData as SD"
]
},
{
"cell_type": "markdown",
"id": "c3914e84",
"metadata": {},
"source": [
"Before starting to create our datasets, we will take a look at the `SampleData` class documenation, to discover the arguments of the class constructor. You can read it on the `pymicro.core` package [API doc page](../../pymicro.core.rst), or print interactively by executing:\n",
"```python\n",
">>> help(SD)\n",
"```\n",
"or, if you are working with a Jupyter notebook, by executing the magic command:\n",
"```\n",
">>> ?SD\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "fff5671f",
"metadata": {},
"source": [
"**Do not hesitate to systematically use the `help` function or the `\"?\"` magic command to get information on methods when you encounter a new one. All SampleData methods are documented with explicative docstrings, that detail the method arguments and returns.**"
]
},
{
"cell_type": "markdown",
"id": "0de71d8d",
"metadata": {},
"source": [
"### Dataset creation"
]
},
{
"cell_type": "markdown",
"id": "bada24e5",
"metadata": {},
"source": [
"The class docstring is divided in multiple rubrics, one of them giving the list of the class constructor arguments. \n",
"Let us review them one by one.\n",
"\n",
"* **filename**: basename of the HDF5 pair of file of the dataset\n",
"\n",
"This is the first and only mandatory argument of the class constructor. If this string corresponds to an existing file, the SampleData class will open these file, and create a file instance to interact with this already existing dataset. **If the filename do not correspond to an existing file, the class will create a new dataset, which is what we want to do here.**\n",
"\n",
"Let us create a SampleData dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c8677d77",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"data = SD(filename='my_first_dataset')"
]
},
{
"cell_type": "markdown",
"id": "3dc87948",
"metadata": {},
"source": [
"That is it. The class has created a new HDF5/XDMF pair of files, and associated the interface with this dataset to the variable `data`. No message has been returned by the code, how can we know that the dataset has been created ?\n",
"\n",
"When the name of the file is not an absolute path, the default behavior of the class is to create the dataset in the current work directory. Let us print the content of this directory then !"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4d3ab02d",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"import os # load python module to interact with operating system\n",
"cwd = os.getcwd() # get current directory\n",
"file_list = os.listdir(cwd) # get content of current work directory\n",
"print(file_list,'\\n')\n",
"\n",
"# now print only HDF5 files\n",
"print('Our dataset files:')\n",
"for file in file_list:\n",
" if file.endswith('.h5'):\n",
" print(file)"
]
},
{
"cell_type": "markdown",
"id": "feab5950",
"metadata": {},
"source": [
"The file *my_first_dataset.h5* has indeed been created. If you want interactive prints about the dataset creation, you can set the **verbose** argument to `True`. This will set the activate the *verbose* mode of the class. When it is, the class instance prints a lot of information about what it is doing. This flag can be set by using the `set_verbosity` method: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8e0cc75d",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"data.set_verbosity(True)"
]
},
{
"cell_type": "markdown",
"id": "42248817",
"metadata": {},
"source": [
"Let us now close our dataset, and see if the class instance prints information about it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "089d50bc",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"del data"
]
},
{
"cell_type": "markdown",
"id": "e9377948",
"metadata": {},
"source": [
"\n",
"\n",
"**Note** \n",
" \n",
"It is a good practice to always delete your `SampleData` instances once you are done working with a dataset, or if you want to re-open it. As the class instance handles opened files as long as it exists, deleting it ensures that the files are properly closed. Otherwise, file may close at some random times or stay opened, and you may encounter undesired behavior of your datasets.\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "d4ca4763",
"metadata": {},
"source": [
"The class indeed returns some prints during the instance destruction. As you can see, the class instance wrights into the HDF5 file the data that is stored into the class instance, and then closes the dataset instance and the files. "
]
},
{
"cell_type": "markdown",
"id": "768b9fc1",
"metadata": {},
"source": [
"### Dataset opening and verbose mode"
]
},
{
"cell_type": "markdown",
"id": "8ee26eeb",
"metadata": {},
"source": [
"Let us now try to create a new SD instance for the same dataset file `\"my_first_dataset\"`. **As the HDF5 dataset already exist, this new *SampleData* instance will open it and synchronize with it.** With the **verbose** mode activated, *SampleData* class instances will display messages about the actions performed by the class (creating, deleting data items for instance)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5fd73ef9",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"data = SD(filename='my_first_dataset', verbose=True)"
]
},
{
"cell_type": "markdown",
"id": "d3483bc2",
"metadata": {},
"source": [
"You can see that the printed information states that the dataset file *my_first_dataset.h5* has been opened, and not created, because we provided a **filename** that already existed to the class constructor.\n",
"\n",
"Some information about the dataset content are also printed by the class in *verbose* mode. This information can be retrived with specific methods that will be detailed in the next section of this Notebook. Let us focus for now on one part of it. \n",
"\n",
"The printed info reveals that our dataset content is composed only of one **data item**, a Group data object named `/`. \n",
"\n",
"This group is the **Root** Group of the dataset. Each dataset has necessarily a Root Group, automatically created along with the dataset. You can see that this Group has no parent group, and already have a *Child*, named `Index`. This particular data object will be presented in the third section of this Notebook. You can also observe that the Root Group already has *attributes* (recall from introduction Notebook that they are Name/Value pairs used to store metadata in datasets). Two of those attributes match arguments of the SampleData class constructor:\n",
"\n",
"\n",
"* the **description** attribute\n",
"* the **sample_name** attribute\n",
"\n",
"**The description and sample_name are not modified in the dataset when reading a dataset. These SD constructor arguments are only used when creating a dataset**. They are string metadata whose role is to give a general name/title to the dataset, and a general description. \n",
"However, they can be set to a new value after the dataset creation with the methods `set_sample_name` and `set_description`, used a little further in this Notebook."
]
},
{
"cell_type": "markdown",
"id": "0b616428",
"metadata": {},
"source": [
"Now we know how to open a dataset previously created with *SampleData*. We could want to open a new dataset, with the name of an already existing data, but overwrite it. The *SampleData* constructor allows to do that, and we will see it in the next subsection. But first, we will close our dataset again: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ff1d4909",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"del data"
]
},
{
"cell_type": "markdown",
"id": "25573424",
"metadata": {},
"source": [
"### Overwriting datasets"
]
},
{
"cell_type": "markdown",
"id": "85214f6a",
"metadata": {},
"source": [
"The **overwrite_hdf5** argument of the class constructor, if it is set to `True`, will remove the `filename` dataset and create a new empty one, if this dataset already exists: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0471a0be",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"data = SD(filename='my_first_dataset', verbose=True, overwrite_hdf5=True)"
]
},
{
"cell_type": "markdown",
"id": "0cbf625c",
"metadata": {},
"source": [
"As you can see, the dataset files have been overwritten, as requested. We will now close our dataset again and continue to see the possibilities offered by the class constructor."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e575b9ae",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"del data"
]
},
{
"cell_type": "markdown",
"id": "bf5a8f23-b9f8-486c-9028-6dd15dada1b8",
"metadata": {},
"source": [
"Our dataset is now closed and we can move on to other ways to create and remove datasets.\n",
" \n",
"**Up to now, there is no mechanism implemented into the class to protect datasets from being overwritten. Be carefull with your data when using this functionality ! **"
]
},
{
"cell_type": "markdown",
"id": "072c2ba2",
"metadata": {},
"source": [
"### Test Copying datasets"
]
},
{
"cell_type": "markdown",
"id": "7c43352f",
"metadata": {},
"source": [
"One last thing that may be interesting to do with already existing dataset files, is to create a new dataset that is a copy of them, associated with a new class instance. This is usefull for instance when you have to try new processing on a set of valuable data, without risking to damage the data. \n",
"\n",
"To do this, you may use the `copy_sample` method of the *SampleData* class. Its main arguments are:\n",
"\n",
"* `src_sample_file`: basename of the dataset files to copy (*source file*)\n",
"* `dst_sample_file`: basename of the dataset to create as a copy of the source (*destination file*)\n",
"* `get_object`: if `False`, the method will just create the new dataset files and close them. If `True`, the method will leave the files open and return a *SampleData* instance that you may use to interact with your new dataset.\n",
"\n",
"Let us try to create a copy of our first dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a4e6e43f",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"data2 = SD.copy_sample(src_sample_file='my_first_dataset', dst_sample_file='dataset_copy', get_object=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6bac2d9c",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"cwd = os.getcwd() # get current directory\n",
"file_list = os.listdir(cwd) # get content of current work directory\n",
"print(file_list,'\\n')\n",
"\n",
"# now print only files that start with our dataset basename\n",
"print('Our dataset files:')\n",
"for file in file_list:\n",
" if file.startswith('dataset_copy'):\n",
" print(file)"
]
},
{
"cell_type": "markdown",
"id": "126a2065",
"metadata": {},
"source": [
"The `copy_dataset.h5` HDF5 file has indeed been created, and is a copy of the `my_first_dataset.h5`.\n",
"\n",
"Note that the `copy_sample` is a *static method*, that can be called even without *SampleData* instance. Note also that it has an `overwrite` argument, that allows to overwrite an already existing `dst_sample_file`. It also has, like the class constructor, a `autodelete` argument, that we will discover in the next subsection."
]
},
{
"cell_type": "markdown",
"id": "6283285f",
"metadata": {},
"source": [
"### Automatically removing dataset files"
]
},
{
"cell_type": "markdown",
"id": "7e9a9d16",
"metadata": {},
"source": [
"In some occasions, we may want to remove our dataset files after using our *SampleData* class instance. This can be the case for instance if you are trying some new data processing, or using the class for visualization purposes, and are not interested in keeping your test data. \n",
"\n",
"The class has a **autodelete** attribute for this purpose. IF it is set to True, the class destructor will remove the dataset file pair in addition to deleting the class instance. The class constructor and the `copy_sample` method also have a **autodelete** argument, which, if `True`, will automatically set the class instance **autodelete** attribute to `True`.\n",
"\n",
"To illustrate this feature, we will try to change the *autodelete* attribute of our copied dataset to `True`, and remove it."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3694a573",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# set the autodelete argument to True\n",
"data2.autodelete = True\n",
"# Set the verbose mode on for copied dataset \n",
"data2.set_verbosity(True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bb6dcefa",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# Close copied dataset\n",
"del data2"
]
},
{
"cell_type": "markdown",
"id": "d5ee91b1",
"metadata": {},
"source": [
"The class destructor ends by priting a confirmation message of the dataset file removal in *verbose* mode, as you can see in the cell above.\n",
"Let us verify that it has been effectively deleted: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9e14e9f2",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"file_list = os.listdir(cwd) # get content of current work directory\n",
"print(file_list,'\\n')\n",
"\n",
"# now print only files that start with our dataset basename\n",
"print('Our copied dataset files:')\n",
"for file in file_list:\n",
" if file.startswith('dataset_copy'):\n",
" print(file)"
]
},
{
"cell_type": "markdown",
"id": "f5c33533",
"metadata": {},
"source": [
"As you can see, the dataset file has been suppressed. Now we can also open and remove our first created dataset using the class constructor **autodelete** option: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2f0cb3f2",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"data = SD(filename='my_first_dataset', verbose=True, autodelete=True)\n",
"\n",
"print(f'Is autodelete mode on ? {data.autodelete}')\n",
"\n",
"del data"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d9ba4849",
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"file_list = os.listdir(cwd) # get content of current work directory\n",
"print(file_list,'\\n')\n",
"\n",
"# now print only files that start with our dataset basename\n",
"print('Our dataset files:')\n",
"for file in file_list:\n",
" if file.startswith('my_first_dataset'):\n",
" print(file)"
]
},
{
"cell_type": "markdown",
"id": "eff1cdb8",
"metadata": {},
"source": [
"\n",
"\n",
"**Note** \n",
" \n",
"Using the **autodelete** option is usefull when you want are using the class for tries, or tests, and do not want to keep the dataset files on your computer. \n",
"\n",
"
"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7a60eadd",
"metadata": {},
"source": [
"**This first tutorial on Data Management with Pymicro User Guide is now finished. You should now know how to create, open or remove SampleData datasets. **"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "env_dev",
"language": "python",
"name": "env_dev"
}
},
"nbformat": 4,
"nbformat_minor": 5
}