Skip to content

Commit 1fe06e0

Browse files
tetiana-karasovakweinmeisterparthea
authored
docs(samples): add product import samples (#149)
* feat: product_number variable is replaced with product_id Import product code samples are added * revert changes in README * fix typo in TEST_RESOURCES_SETUP_CLEANUP.md * include project number in import_products_big_query_table.py * use project_number in import_products_gcs.py; use f-string * use project_number in import_products_inline_source.py * use project_number in product/setup/setup_cleanup.py * use project_number in search_simple_query.py * use project_number in search_with_boost_spec.py * use project_number in search_with_filtering.py * use project number in search_with_ordering.py * use project_number in search_with_pagination.py * use project_number in search_with_query_expansion_spec.py * use project_number in test_resources_recovery * use project_number in search_with_facet_spec.py * fix typo in f-string * fix typos * use correct path to resources * lint * revert change to paths * resolve error where bq table doesn't exist * make setup more robust * lint * import product from inline source is fixed * use google.cloud.bigquery client * update test to reflect changes * lint Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
1 parent 2f7f95d commit 1fe06e0

20 files changed

Lines changed: 1167 additions & 23 deletions

‎generated_samples/interactive-tutorials/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -72,19 +72,19 @@ To run a code sample from the Cloud Shell, you need to be authenticated using th
7272
export GOOGLE_APPLICATION_CREDENTIALS=~/key.json
7373
```
7474

75-
### Set the PROJECT_NUMBER and PROJECT_ID environment variables
75+
### Set the GOOGLE_CLOUD_PROJECT_NUMBER and GOOGLE_CLOUD_PROJECT environment variables
7676

7777
You will run the code samples in your own Google Cloud project. To use the **project_number** and **project_id** in every request to the Retail API, you should first specify them as environment variables.
7878

7979
1. Find the project number and project ID in the Project Info card displayed on **Home/Dashboard**.
8080

8181
1. Set the **project_number** with the following command:
8282
```bash
83-
export PROJECT_NUMBER=<YOUR_PROJECT_NUMBER>
83+
export GOOGLE_CLOUD_PROJECT_NUMBER=<YOUR_PROJECT_NUMBER>
8484
```
8585
1. Set the **project_id** with the following command:
8686
```bash
87-
export PROJECT_ID=<YOUR_PROJECT_ID>
87+
export GOOGLE_CLOUD_PROJECT=<YOUR_PROJECT_ID>
8888
```
8989

9090
### Install Google Cloud Retail libraries

‎generated_samples/interactive-tutorials/TEST_RESOURCES_SETUP_CLEANUP.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
To successfully import the catalog data for tests, the following environment variables should be set:
66
- GOOGLE_CLOUD_PROJECT_NUMBER
7-
- GOOGLE_CLOUD_PROJECT_ID
7+
- GOOGLE_CLOUD_PROJECT
88
- BUCKET_NAME
99
- EVENTS_BUCKET_NAME
1010
These values are stored in the Secret Manager and will be submitted as

‎generated_samples/interactive-tutorials/product/import_products_big_query_table.py‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Copyright 2021 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# [START retail_import_products_from_big_query]
16+
# Import products into a catalog from big query table using Retail API
17+
#
18+
import os
19+
import time
20+
21+
from google.cloud.retail import (
22+
BigQuerySource,
23+
ImportProductsRequest,
24+
ProductInputConfig,
25+
ProductServiceClient,
26+
)
27+
28+
project_number = os.environ["GOOGLE_CLOUD_PROJECT_NUMBER"]
29+
project_id = os.environ["GOOGLE_CLOUD_PROJECT"]
30+
31+
default_catalog = f"projects/{project_number}/locations/global/catalogs/default_catalog/branches/default_branch"
32+
dataset_id = "products"
33+
table_id = "products"
34+
35+
36+
# TO CHECK ERROR HANDLING USE THE TABLE WITH INVALID PRODUCTS:
37+
# table_id = "products_some_invalid"
38+
39+
40+
# get import products from big query request
41+
def get_import_products_big_query_request(reconciliation_mode):
42+
# TO CHECK ERROR HANDLING PASTE THE INVALID CATALOG NAME HERE:
43+
# default_catalog = "invalid_catalog_name"
44+
big_query_source = BigQuerySource()
45+
big_query_source.project_id = project_id
46+
big_query_source.dataset_id = dataset_id
47+
big_query_source.table_id = table_id
48+
big_query_source.data_schema = "product"
49+
50+
input_config = ProductInputConfig()
51+
input_config.big_query_source = big_query_source
52+
53+
import_request = ImportProductsRequest()
54+
import_request.parent = default_catalog
55+
import_request.reconciliation_mode = reconciliation_mode
56+
import_request.input_config = input_config
57+
58+
print("---import products from big query table request---")
59+
print(import_request)
60+
61+
return import_request
62+
63+
64+
# call the Retail API to import products
65+
def import_products_from_big_query():
66+
# TRY THE FULL RECONCILIATION MODE HERE:
67+
reconciliation_mode = ImportProductsRequest.ReconciliationMode.INCREMENTAL
68+
69+
import_big_query_request = get_import_products_big_query_request(
70+
reconciliation_mode
71+
)
72+
big_query_operation = ProductServiceClient().import_products(
73+
import_big_query_request
74+
)
75+
76+
print("---the operation was started:----")
77+
print(big_query_operation.operation.name)
78+
79+
while not big_query_operation.done():
80+
print("---please wait till operation is done---")
81+
time.sleep(30)
82+
print("---import products operation is done---")
83+
84+
if big_query_operation.metadata is not None:
85+
print("---number of successfully imported products---")
86+
print(big_query_operation.metadata.success_count)
87+
print("---number of failures during the importing---")
88+
print(big_query_operation.metadata.failure_count)
89+
else:
90+
print("---operation.metadata is empty---")
91+
92+
if big_query_operation.result is not None:
93+
print("---operation result:---")
94+
print(big_query_operation.result())
95+
else:
96+
print("---operation.result is empty---")
97+
98+
99+
import_products_from_big_query()
100+
101+
# [END retail_import_products_from_big_query]

‎generated_samples/interactive-tutorials/product/import_products_bq_test.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Copyright 2021 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import re
16+
import subprocess
17+
18+
19+
def test_import_products_bq():
20+
output = str(
21+
subprocess.check_output("python import_products_big_query_table.py", shell=True)
22+
)
23+
24+
assert re.match(".*import products from big query table request.*", output)
25+
assert re.match(".*the operation was started.*", output)
26+
assert re.match(
27+
".*projects/.*/locations/global/catalogs/default_catalog/branches/0/operations/import-products.*",
28+
output,
29+
)
30+
31+
assert re.match(".*number of successfully imported products.*316.*", output)

‎generated_samples/interactive-tutorials/product/import_products_gcs.py‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Copyright 2021 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# [START retail_import_products_from_gcs]
16+
# Import products into a catalog from gcs using Retail API
17+
#
18+
import os
19+
import time
20+
21+
from google.cloud.retail import (
22+
GcsSource,
23+
ImportErrorsConfig,
24+
ImportProductsRequest,
25+
ProductInputConfig,
26+
ProductServiceClient,
27+
)
28+
29+
30+
# Read the project number from the environment variable
31+
project_number = os.environ["GOOGLE_CLOUD_PROJECT_NUMBER"]
32+
project_id = os.environ["GOOGLE_CLOUD_PROJECT"]
33+
bucket_name = os.environ["BUCKET_NAME"]
34+
35+
# You can change the branch here. The "default_branch" is set to point to the branch "0"
36+
default_catalog = f"projects/{project_number}/locations/global/catalogs/default_catalog/branches/default_branch"
37+
38+
gcs_bucket = f"gs://{bucket_name}"
39+
gcs_errors_bucket = f"{gcs_bucket}/error"
40+
gcs_products_object = "products.json"
41+
42+
43+
# TO CHECK ERROR HANDLING USE THE JSON WITH INVALID PRODUCT
44+
# gcs_products_object = "products_some_invalid.json"
45+
46+
47+
# get import products from gcs request
48+
def get_import_products_gcs_request(gcs_object_name: str):
49+
# TO CHECK ERROR HANDLING PASTE THE INVALID CATALOG NAME HERE:
50+
# default_catalog = "invalid_catalog_name"
51+
gcs_source = GcsSource()
52+
gcs_source.input_uris = [f"{gcs_bucket}/{gcs_object_name}"]
53+
54+
input_config = ProductInputConfig()
55+
input_config.gcs_source = gcs_source
56+
print("GRS source:")
57+
print(gcs_source.input_uris)
58+
59+
errors_config = ImportErrorsConfig()
60+
errors_config.gcs_prefix = gcs_errors_bucket
61+
62+
import_request = ImportProductsRequest()
63+
import_request.parent = default_catalog
64+
import_request.reconciliation_mode = (
65+
ImportProductsRequest.ReconciliationMode.INCREMENTAL
66+
)
67+
import_request.input_config = input_config
68+
import_request.errors_config = errors_config
69+
70+
print("---import products from google cloud source request---")
71+
print(import_request)
72+
73+
return import_request
74+
75+
76+
# call the Retail API to import products
77+
def import_products_from_gcs():
78+
import_gcs_request = get_import_products_gcs_request(gcs_products_object)
79+
gcs_operation = ProductServiceClient().import_products(import_gcs_request)
80+
81+
print("---the operation was started:----")
82+
print(gcs_operation.operation.name)
83+
84+
while not gcs_operation.done():
85+
print("---please wait till operation is done---")
86+
time.sleep(30)
87+
print("---import products operation is done---")
88+
89+
if gcs_operation.metadata is not None:
90+
print("---number of successfully imported products---")
91+
print(gcs_operation.metadata.success_count)
92+
print("---number of failures during the importing---")
93+
print(gcs_operation.metadata.failure_count)
94+
else:
95+
print("---operation.metadata is empty---")
96+
97+
if gcs_operation.result is not None:
98+
print("---operation result:---")
99+
print(gcs_operation.result())
100+
else:
101+
print("---operation.result is empty---")
102+
103+
# The imported products needs to be indexed in the catalog before they become available for search.
104+
print(
105+
"Wait 2-5 minutes till products become indexed in the catalog, after that they will be available for search"
106+
)
107+
108+
109+
import_products_from_gcs()
110+
111+
# [END retail_import_products_from_gcs]

‎generated_samples/interactive-tutorials/product/import_products_gcs_test.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Copyright 2021 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import re
16+
import subprocess
17+
18+
19+
def test_import_products_gcs():
20+
output = str(subprocess.check_output("python import_products_gcs.py", shell=True))
21+
22+
assert re.match(".*import products from google cloud source request.*", output)
23+
assert re.match('.*input_uris: "gs://.*/products.json".*', output)
24+
assert re.match(".*the operation was started.*", output)
25+
assert re.match(
26+
".*projects/.*/locations/global/catalogs/default_catalog/branches/0/operations/import-products.*",
27+
output,
28+
)
29+
30+
assert re.match(".*number of successfully imported products.*316.*", output)

0 commit comments

Comments
 (0)