Pitfalls of Passing Scope Across Services in Python

Background In an old system, there is a piece of code like this: scope = dict(globals(), **locals()) exec( """ global_a = 123 def func_a(): print(global_a) """ , scope) exec("func_a()", scope) The first segment of user code defines a function, and the second segment executes the function (don’t ask why this is done, because the user is always right). After executing the first code block, func_a and global_a are added to the scope. Since the second code block uses the same scope, calling func_a in the second block correctly outputs 123. ...

November 6, 2021 · 4 min · Zhiya

Using AWS Lambda and iOS Shortcuts for One-Tap Access to Community Gates

The residential community where I live uses a smart access control system called “Watchful Domain (守望领域)” which allows opening community and unit gates via a mobile app. However, using the app to open gates involves several steps: open the app → navigate to the gate control interface → find the gate you want to open → tap to open. Moreover, unlocking the phone with a mask on requires entering a password, making the process quite time-consuming. Often, I find myself standing at the community or unit entrance for a while. At one point, I even got into the habit of carrying a physical access card, which is much quicker for opening gates. ...

October 19, 2021 · 8 min · Zhiya

Issues Arising from Cross-Process Use of gRPC

Problem Description When using gRPC for communication in a Python project, issues such as blocking or errors can occur when used across processes (the symptoms vary depending on the version of gRPC.io). The following code demonstrates a cross-process usage demo where the main process sends requests to a gRPC server on port 30001, and the child process also sends requests to the same server. def send(): channel = grpc.insecure_channel('localhost:30001') stub = message_pb2_grpc.GreeterStub(channel) response = stub.SayHello(message_pb2.HelloRequest(name='you')) print(f"Greeter client received 1: " + response.message) def main(): channel = grpc.insecure_channel('localhost:30001') stub = message_pb2_grpc.GreeterStub(channel) response = stub.SayHello2(message_pb2.HelloRequest(name='you')) print("Greeter client received 2: " + response.message) p = multiprocessing.Process(target=send) p.start() p.join() if __name__ == '__main__': main() When using gRPC.io version 1.28.1, an error occurs: the main process can receive the server’s response normally, but the child process reports Socket operation on non-socket. ...

April 23, 2020 · 4 min · Zhiya

Using Pipfile Instead of requirements.txt

Many programming languages provide support for environment isolation, such as Node.js with node_modules, Golang with go mod, and Python with virtualenv and pyvenv. To create a dependency snapshot, the pip freeze > requirements.txt command is often used to generate a requirements.txt file. In some scenarios, this method suffices, but in more complex situations, requirements.txt falls short. requirements.txt appdirs==1.4.3 astroid==2.3.3 attrs==19.3.0 black==19.3b0 certifi==2019.11.28 chardet==3.0.4 click==7.1.1 et-xmlfile==1.0.1 Flask==1.1.1 gevent==1.4.0 greenlet==0.4.15 idna==2.9 isort==4.3.21 itsdangerous==1.1.0 jdcal==1.4.1 Jinja2==2.11.1 lazy-object-proxy==1.4.3 MarkupSafe==1.1.1 mccabe==0.6.1 numpy==1.18.2 openpyxl==3.0.3 pandas==1.0.3 pylint==2.4.4 python-dateutil==2.8.1 pytz==2019.3 requests==2.23.0 six==1.14.0 tinydb==3.15.2 toml==0.10.0 typed-ast==1.4.1 urllib3==1.25.8 Werkzeug==1.0.0 wrapt==1.11.2 The requirements.txt file only records the versions of dependencies. If the official PyPI source is slow, and you need to use a faster domestic mirror, you typically have to use pip install -i or modify the global pip.conf file. ...

March 31, 2020 · 3 min · Zhiya

Exploring Pandas Read Excel File Error

Problem Description Using Pandas’ read_excel method to read an Excel file with 160,000 rows results in an AssertionError: "/Users/XXX/excel_test/venv/lib/python3.7/site-packages/xlrd/xlsx.py", line 637, in do_row assert 0 <= self.rowx < X12_MAX_ROWS AssertionError Underlying Principle Excel files come in two default formats. Before Excel 2007, files used the .xls format, a specific binary format supporting up to 65,536 rows (16,384 rows before Excel 97) and 256 columns. Starting with Excel 2007, a new XML-based format .xlsx was adopted, supporting up to 1,048,576 rows and 16,384 columns. Note that when converting a .xlsx file to .xls, data beyond 65,536 rows and 256 columns will be lost. ...

August 22, 2019 · 3 min · Zhiya