Skip to content

Commit ebfd167

Browse files
yifanfeng97yifanfeng97
committed
update doc tutorial dataset
1 parent b240739 commit ebfd167

5 files changed

Lines changed: 537 additions & 40 deletions

File tree

‎dhg/data/base.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ class BaseData:
1212
::
1313
1414
self._content = {
15-
'feature': {
15+
'item': {
1616
'upon': [
1717
{'filename': 'part1.pkl', 'md5': '', bk_url: None},
1818
{'filename': 'part2.pkl', 'md5': '', bk_url: None},

‎docs/source/api/datapipe.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Compose Datapipes
88

99
.. autofunction:: dhg.datapipe.compose_pipes
1010

11+
.. _api_datapipe_preprocess:
1112

1213
Transforms
1314
--------------
@@ -21,6 +22,8 @@ Transforms
2122
.. autofunction:: dhg.datapipe.to_long_tensor
2223

2324

25+
.. _api_datapipe_loader:
26+
2427
Loaders
2528
---------
2629

‎docs/source/tutorial/dataset.rst‎

Lines changed: 191 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ please following the :doc:`instruction </start/contribution>` guide.
1010
Usage
1111
-----------------------
1212

13-
If your network is OK, you can directly use any of datasets in :doc:`/api/data` as following:
13+
If your network is OK, you can directly use any of the datasets in :doc:`/api/data` as follows:
1414

1515
.. code-block:: python
1616
@@ -149,36 +149,208 @@ The architecture of constructing DHG's dataset object is shown in the following
149149

150150
Build Your Own Dataset
151151
-----------------------
152-
Coming soon...
153152

153+
At first you should inherit your data class from the :py:class:`BaseData <dhg.data.BaseData>` class.
154154

155+
.. code-block:: python
156+
157+
>>> from dhg.data import BaseData
158+
159+
All the items in the dataset are configured in the ``_content`` dictionary. Currently, the following items are supported:
160+
161+
- download from remote server -> load from local file -> preprocess and return
162+
- load from local file -> preprocess and return
163+
- directly return
155164

156-
.. Prepare Dataset
157-
.. -----------------
165+
The supported loader functions can be found in :ref:`here <api_datapipe_loader>`.
158166

159-
.. Use the intergrated dataset
160-
.. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
167+
The supported preprocess functions can be found in :ref:`here <api_datapipe_preprocess>`.
161168

162-
.. Currently, DHG includes the following datasets:
169+
If the ``item`` should be downloaded from a remote server, you should specify the ``upon``, ``loader``, and ``preprocess`` keys in the ``_content`` dictionary.
170+
The ``upon`` key is a list of dictionaries, each dictionary at lease contains the ``filename`` and ``md5`` keys.
171+
The ``filename`` is the name of the file to be downloaded, and the ``md5`` is the md5 checksum of the file.
172+
Defaultly, remote file is stored in the ``REMOTE_DATASETS_ROOT \ data_root \ name \ filename`` directory.
163173

174+
.. code-block:: python
175+
176+
self._content = {
177+
'item': {
178+
'upon': [
179+
{'filename': 'part1.pkl', 'md5': '', bk_url: None},
180+
{'filename': 'part2.pkl', 'md5': '', bk_url: None},
181+
],
182+
'loader': loader_function,
183+
'preprocess': [datapipe1, datapipe2],
184+
},
185+
...
186+
}
164187
165188
166-
.. Dataset Pipeline
167-
.. ------------------
189+
If the ``item`` relay on a local file, you should also specify the ``upon``, ``loader``, and ``preprocess`` keys in the ``_content`` dictionary.
190+
But the file should be put into the ``data_root \ name \ filename`` directory.
191+
Then, the :py:class:`BaseData <dhg.data.BaseData>` class will automatically check the file's md5 checksum.
168192

169-
.. How to process the data
193+
.. code-block:: python
194+
195+
self._content = {
196+
'item': {
197+
'upon': [
198+
{'filename': 'part1.pkl', 'md5': '', bk_url: None},
199+
{'filename': 'part2.pkl', 'md5': '', bk_url: None},
200+
],
201+
'loader': loader_function,
202+
'preprocess': [datapipe1, datapipe2],
203+
},
204+
...
205+
}
206+
207+
If the ``item`` is a fixed value, you can directly specify the ``value`` in the ``_content`` dictionary.
170208

171-
.. Available Pipeline Functions
172-
.. -----------------------------
209+
.. code-block:: python
210+
211+
self._content = {
212+
'item': 666666,
213+
...
214+
}
173215
174-
.. to_tensor
175216
176-
.. Introduction
177-
.. ------------------------
178-
.. For each dataset in DHG, we have pre-process the feature. and transform them to torch.Tensor.
217+
Example of Graph Dataset:
218+
++++++++++++++++++++++++++++
179219

180-
.. You can access the raw data by data.raw('attribute_name')
220+
.. code-block:: python
221+
222+
class Cora(BaseData):
223+
def __init__(self, data_root: Optional[str] = None) -> None:
224+
super().__init__('cora', data_root)
225+
self._content = {
226+
"num_classes": 7,
227+
"num_vertices": 2708,
228+
"num_edges": 10858,
229+
"dim_features": 1433,
230+
'features': {
231+
'upon': [{ 'filename': 'features.pkl', 'md5': '05b45e9c38cc95f4fc44b3668cc9ddc9' }],
232+
'loader': load_from_pickle,
233+
'preprocess': [to_tensor, partial(norm_ft, ord=1)],
234+
},
235+
'edge_list': {
236+
'upon': [{ 'filename': 'edge_list.pkl', 'md5': 'f488389c1edd0d898ce273fbd27822b3' }],
237+
'loader': load_from_pickle,
238+
},
239+
'labels': {
240+
'upon': [{ 'filename': 'labels.pkl', 'md5': 'e506014762052c6a36cb583c28bdae1d' }],
241+
'loader': load_from_pickle,
242+
'preprocess': [to_long_tensor],
243+
},
244+
'train_mask': {
245+
'upon': [{ 'filename': 'train_mask.pkl', 'md5': 'a11357a40e1f0b5cce728d1a961b8e13' }],
246+
'loader': load_from_pickle,
247+
'preprocess': [to_bool_tensor],
248+
},
249+
'val_mask': {
250+
'upon': [{ 'filename': 'val_mask.pkl', 'md5': '355544da566452601bcfa74d30539a71' }],
251+
'loader': load_from_pickle,
252+
'preprocess': [to_bool_tensor],
253+
},
254+
'test_mask': {
255+
'upon': [{ 'filename': 'test_mask.pkl', 'md5': 'bbfc87d661560f55f6946f8cb9d602b9' }],
256+
'loader': load_from_pickle,
257+
'preprocess': [to_bool_tensor],
258+
},
259+
}
260+
261+
Example of Hypergraph Dataset
262+
++++++++++++++++++++++++++++++++
263+
264+
.. code-block:: python
265+
266+
class Cooking200(BaseData):
267+
def __init__(self, data_root: Optional[str] = None) -> None:
268+
super().__init__("cooking_200", data_root)
269+
self._content = {
270+
"num_classes": 20,
271+
"num_vertices": 7403,
272+
"num_edges": 2755,
273+
"edge_list": {
274+
"upon": [
275+
{
276+
"filename": "edge_list.pkl",
277+
"md5": "2cd32e13dd4e33576c43936542975220",
278+
}
279+
],
280+
"loader": load_from_pickle,
281+
},
282+
"labels": {
283+
"upon": [
284+
{
285+
"filename": "labels.pkl",
286+
"md5": "f1f3c0399c9c28547088f44e0bfd5c81",
287+
}
288+
],
289+
"loader": load_from_pickle,
290+
"preprocess": [to_long_tensor],
291+
},
292+
"train_mask": {
293+
"upon": [
294+
{
295+
"filename": "train_mask.pkl",
296+
"md5": "66ea36bae024aaaed289e1998fe894bd",
297+
}
298+
],
299+
"loader": load_from_pickle,
300+
"preprocess": [to_bool_tensor],
301+
},
302+
"val_mask": {
303+
"upon": [
304+
{
305+
"filename": "val_mask.pkl",
306+
"md5": "6c0d3d8b752e3955c64788cc65dcd018",
307+
}
308+
],
309+
"loader": load_from_pickle,
310+
"preprocess": [to_bool_tensor],
311+
},
312+
"test_mask": {
313+
"upon": [
314+
{
315+
"filename": "test_mask.pkl",
316+
"md5": "0e1564904551ba493e1f8a09d103461e",
317+
}
318+
],
319+
"loader": load_from_pickle,
320+
"preprocess": [to_bool_tensor],
321+
},
322+
}
323+
324+
325+
Example of User-Item Bipartite Dataset
326+
++++++++++++++++++++++++++++++++++++++++++++
327+
328+
.. code-block:: python
181329
330+
class MovieLens1M(BaseData):
331+
def __init__(self, data_root: Optional[str] = None) -> None:
332+
super().__init__("movielens_1m", data_root)
333+
self._content = {
334+
"num_users": 6022,
335+
"num_items": 3043,
336+
"num_interactions": 995154,
337+
"train_adj_list": {
338+
"upon": [
339+
{
340+
"filename": "train.txt",
341+
"md5": "db93f671bc5d1b1544ce4c29664f6778",
342+
}
343+
],
344+
"loader": partial(load_from_txt, dtype="int", sep=" "),
345+
},
346+
"test_adj_list": {
347+
"upon": [
348+
{
349+
"filename": "test.txt",
350+
"md5": "5e55bcbb6372ad4c6fafe79989e2f956",
351+
}
352+
],
353+
"loader": partial(load_from_txt, dtype="int", sep=" "),
354+
},
355+
}
182356
183-
.. Examples
184-
.. --------------

‎docs/source/tutorial/structure.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,8 +59,8 @@ Common Methods
5959
[0., 0., 0., 1., 0.]])
6060
6161
You can find that the adjacency matrix of the simple graph is a symmetric matrix.
62-
The ``g.e`` attribute will return a tuple of two lists, the first list is the edge list and the second list is a list of weight for each edge.
63-
The ``g.e_both_size`` attribute will return the both side of edges in the simple graph.
62+
The :py:attr:`g.e <dhg.Graph.e>` attribute will return a tuple of two lists, the first list is the edge list and the second list is a list of weight for each edge.
63+
The :py:attr:`g.e_both_side <dhg.Graph.e_both_side>` attribute will return the both side of edges in the simple graph.
6464

6565
.. important::
6666

0 commit comments

Comments
 (0)