No packages installed when using requirements.txt from pip-tools

[brettdh]

Our project uses pip-tools to handle dependency pinning. This produces a requirements.txt that looks something like this:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes --output-file requirements.txt requirements.in
#
aioboto3==4.0.2     \
    --hash=sha256:33025e8143b56a57d36fdbf9aa6fea90c5811c33b4a318d837117ba8c5e1ffee     \
    --hash=sha256:a3ff295154a00a4bc609d1e50df8abb5b1266ec357c9fe2003a94c7e62c9548e
aiobotocore==0.9.3     \
    --hash=sha256:d4937d51caf94ad124ff68303a859c25a777a4a124f5e687b8187476e056b101     \
    --hash=sha256:f8547a58dc4e81ff62ab2d1f15fe730dcd26476ff076c1b29a38dc1a598ef2f4     \
    # via aioboto3
...

and so on.

However, when serverless-python-requirements processes this file, it produces a requirements.txt that looks like this:

--hash=sha256:33025e8143b56a57d36fdbf9aa6fea90c5811c33b4a318d837117ba8c5e1ffee     \
--hash=sha256:a3ff295154a00a4bc609d1e50df8abb5b1266ec357c9fe2003a94c7e62c9548e
--hash=sha256:d4937d51caf94ad124ff68303a859c25a777a4a124f5e687b8187476e056b101     \
--hash=sha256:f8547a58dc4e81ff62ab2d1f15fe730dcd26476ff076c1b29a38dc1a598ef2f4     \
aioboto3==4.0.2     \
aiobotocore==0.9.3     \

which ultimately results in no packages being installed in ./serverless/requirements/ and thus nothing being injected into the deployment .zip.

I've worked around this by having a wrapper script around pip-compile which strips out the escaped newlines and joins the separated lines, producing a requirements.txt more akin to what Pipenv generates. I'd prefer not to have to resort to this hack, though.

---

Additionally, the plugin's default behavior of stripping boto3 and its dependencies led to surprising behavior:

  In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    botocore<1.10.59,>=1.10.58 from https://files.pythonhosted.org/packages/cb/6e/4d961f0c98007ba077a64626f4e0056e5351cca54653b15d12bb03c32110/botocore-1.10.58-py2.py3-none-any.whl#sha256=17a88a578161dc12ecf14950afa93a354cf009380977921f7f
52891acc5e751a (from aioboto3==4.0.2->-r /var/task/requirements.txt (line 1))
    boto3<=1.7.58,>=1.4.7 from https://files.pythonhosted.org/packages/b8/7c/f7213ba8cf2b660a6c0cdf055290e6df71037a60fc1db3d8e4a79b4789ee/boto3-1.7.58-py2.py3-none-any.whl#sha256=08f268d6eb3347061384e144121dcca1e454a7a8b6c8424a23d3a312cde
bab68 (from aioboto3==4.0.2->-r /var/task/requirements.txt (line 1))
    six from https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl#sha256=832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb (from djangore
stframework-jsonapi==2.3.1->-r /var/task/requirements.txt (line 12))
    python-dateutil>=2.6.0 from https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl#sha256=1adb80e7a782c12e52ef9a8182bebeb73f1d7e24e374397af
06fb4956c8dc5c0 (from influxdb==5.0.0->-r /var/task/requirements.txt (line 18))
You are using pip version 9.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I've added a bogus noDeploy: ['nonexistent'] configuration to avoid this (even though there's nothing I actually need to exclude), but I'd prefer it if the plugin was either smarter about removing dependencies under the covers, or else a bit dumber about it (i.e. just deploy what's in my requirements.txt by default; don't strip things even though they might be redundant).

Though this latter problem might seem like it merits a separate issue, I've included it here as it seems like a similar example of the plugin doing a bit too much in the way of mangling the requirements.txt that I provide.

[dschep]

Thanks for this bug report. I'll take a look at it this evening.. Re your noDeploy: ['nonexistant'] you can also just do noDeploy: []