pull-through-caching not work as expected

[milad-pirooz]

Version
i installed the latest tags (pulpcore 3.81.0, pulp_python 3.16.0) via docker compose

Describe the bug
i created a pull-through-caching distribution like this:

pulp python repository create --name pypi
pulp python remote create --name pypiorg --url https://pypi.org/
pulp python distribution create --name pypi --base-path pypi --repository pypi --remote pypiorg

and then trying to install django package:

pip install django==5.1 -i http://127.0.0.1:8080/pulp/content/pypi/simple --trusted-host 127.0.0.1

the pip installation is ok, but in the configured S3 storate, i get just html files stored, containing package index pages
links in the htnl index, points to files.pythonhosted.org, not the content server.
also in pulp content logs, i get:

pulp_content_1               | Traceback (most recent call last):
pulp_content_1               |   File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 510, in _handle_request
pulp_content_1               |     resp = await request_handler(request)
pulp_content_1               |   File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py", line 569, in _handle
pulp_content_1               |     return await handler(request)
pulp_content_1               |   File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
pulp_content_1               |     return await handler(request)
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 27, in guid
pulp_content_1               |     return await handler(request)
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 58, in authenticate
pulp_content_1               |     return await handler(request)
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 354, in cached_function
pulp_content_1               |     response = await self.make_entry(
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 400, in make_entry
pulp_content_1               |     response = await handler(*args, **kwargs)
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 290, in stream_content
pulp_content_1               |     return await self._match_and_stream(path, request)
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 918, in _match_and_stream
pulp_content_1               |     return await self._stream_remote_artifact(
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 1334, in _stream_remote_artifact
pulp_content_1               |     content_artifacts = await asyncio.shield(
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 468, in __call__
pulp_content_1               |     ret = await asyncio.shield(exec_coro)
pulp_content_1               |   File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
pulp_content_1               |     result = self.fn(*self.args, **self.kwargs)
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 522, in thread_handler
pulp_content_1               |     return func(*args, **kwargs)
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 1048, in _save_artifact

pulp_content_1               |     content = c_type.init_from_artifact_and_relative_path(artifact, rel_path)
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulp_python/app/models.py", line 191, in init_from_artifact_and_relative_path
pulp_content_1               |     data = artifact_to_python_content_data(path.name, artifact, domain=get_domain())
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulp_python/app/utils.py", line 184, in artifact_to_python_content_data
pulp_content_1               |     metadata = get_project_metadata_from_artifact(filename, artifact)
pulp_content_1               |   File "/usr/local/lib/python3.9/site-packages/pulp_python/app/utils.py", line 159, in get_project_metadata_from_artifact
pulp_content_1               |     pkg_type_index = [filename.endswith(ext) for ext in extensions].index(True)
pulp_content_1               | ValueError: True is not in list

To Reproduce
Steps to reproduce the behavior:
just configure the storage in settings.py, and do docker-compose up -d

Expected behavior
to store the django package and its dependencies in configured S3 backend on demand.

[jobselko]

@milad-pirooz You need to use pypi instead of pulp/content, so the resulted url used in pip install should be http://127.0.0.1:8080/pypi/pypi/simple (see https://github.com/pulp/pulp_python/blob/main/pulp_python/app/serializers.py#L62).

When using pulp/content (which is originally from pulpcore), the pull-through cache is not used correctly. This is actually a bug, so I am leaving the issue open to track it.