test_zipfile fails on AIX due to time.localtime

[EGuesnet]

BPO	39502
Nosy	@vstinner, @aixtools, @miss-islington, @EGuesnet
PRs	bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX #18282 [3.8] bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282) #18283 bpo-39502: Fix 64-bit Python PyTime_localtime() on AIX  #18285
Files	test_zipfile.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-30.13:51:37.691>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.7', 'tests']
title = 'test_zipfile fails on AIX due to time.localtime'
updated_at = <Date 2020-03-26.09:42:37.096>
user = 'https://github.com/EGuesnet'

bugs.python.org fields:

activity = <Date 2020-03-26.09:42:37.096>
actor = 'Michael.Felt'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Tests']
creation = <Date 2020-01-30.13:51:37.691>
creator = 'EGuesnet'
dependencies = []
files = ['48872']
hgrepos = []
issue_num = 39502
keywords = ['patch']
message_count = 21.0
messages = ['361058', '361059', '361061', '361062', '361063', '361064', '361065', '361066', '361067', '361068', '361069', '361070', '361071', '361116', '361118', '361272', '361303', '361323', '361603', '361604', '365056']
nosy_count = 5.0
nosy_names = ['vstinner', 'David.Edelsohn', 'Michael.Felt', 'miss-islington', 'EGuesnet']
pr_nums = ['18282', '18283', '18285']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39502'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[EGuesnet]

Hi,
I have an error during regression tests with Python3.8.1 on AIX 6.1 compiled with GCC 8.3.
It occurs only on 64 bit. Test passes on 32 bit.

======================================================================
ERROR: test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/test/test_zipfile.py", line 606, in test_add_file_after_2107
    self.assertRaises(struct.error, zipfp.write, TESTFN)
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/unittest/case.py", line 816, in assertRaises
    return context.handle('assertRaises', args, kwargs)
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/unittest/case.py", line 202, in handle
    callable_obj(*args, **kwargs)
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/zipfile.py", line 1739, in write
    zinfo = ZipInfo.from_file(filename, arcname,
  File "/opt/freeware/src/packages/BUILD/Python-3.8.1/64bit/Lib/zipfile.py", line 523, in from_file
    mtime = time.localtime(st.st_mtime)
OverflowError: localtime argument out of range

The PR associated to the new behavior is: #12726 (new on Python 3.8). Code is AIX specific. Is the code 32 bit only, or maybe the test was not updated?

-----

I can reproduce the behavior as follow:

$ python3.8_32
Python 3.8.1 (default, Jan 27 2020, 11:34:59) 
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

$ python3.8_64
Python 3.8.1 (default, Jan 27 2020, 11:30:15) 
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: localtime argument out of range

$ python3.7_32
Python 3.7.4 (default, Jan 15 2020, 15:50:53) 
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

$ python3.7_64
Python 3.7.4 (default, Jan 15 2020, 15:46:22) 
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.localtime(4325562452)
time.struct_time(tm_year=2107, tm_mon=1, tm_mday=27, tm_hour=10, tm_min=7, tm_sec=32, tm_wday=3, tm_yday=27, tm_isdst=0)

[vstinner]

I suggest to skip the test if time.localtime() fails with OverflowError.

Can you please try attached patch on AIX, on the master branch if possible? (Or on Python 3.8 at least.)

[EGuesnet]

Tested on Python 3.8.1 on both 32 and 64 bits.
Test is now skipped.

test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'time.localtime(4386268800) raises OverflowError'

[vstinner]

Tested on Python 3.8.1 on both 32 and 64 bits. Test is now skipped.

Good. I created PR 18282 from my patch.

[aixtools]

Probably this broke the 64-bit usage.

diff --git a/Python/pytime.c b/Python/pytime.c
index 54ddfc952b..6f13e62490 100644
--- a/Python/pytime.c
+++ b/Python/pytime.c
@@ -1059,7 +1059,7 @@ _PyTime_localtime(time_t t, struct tm *tm)
return 0;
#else /* !MS_WINDOWS */

-#ifdef _AIX
+#ifdef defined(_AIX) and (SIZEOF_LONG ==4)
     /* bpo-34373: AIX does not return NULL if t is too small or too large */
     if (t < -2145916800 /* 1902-01-01 */
        || t > 2145916800 /* 2038-01-01 */) {

Testing...

[vstinner]

Probably this broke the 64-bit usage.

I'm not sure of the meaning of your patch. Are you saying that localtime() supports timestamp after the year 2038 on 64-bit AIX? Did you test that time.localtime(2**32) actually works as expected?

In my timezone, Linux, I get:

$ python3
Python 3.7.6 (default, Dec 19 2019, 22:52:49) 
>>> import time; time.localtime(2**32)
time.struct_time(tm_year=2106, tm_mon=2, tm_mday=7, tm_hour=7, tm_min=28, tm_sec=16, tm_wday=6, tm_yday=38, tm_isdst=0)

[aixtools]

From memory I recall the 64-bit version worked with values above the threshold value that broke the 32-bit library.

And the additional test was needed because the AIX library (iirc did not return NULL on error) - so had to test range before the call to get an overflow.

Working up a test of 32 and 64-bit enviornments.

[vstinner]

New changeset c232c91 by Victor Stinner in branch 'master':
bpo-39502: Skip test_zipfile.test_add_file_after_2107() on AIX (GH-18282)
c232c91

[EGuesnet]

I'm not sure of the meaning of your patch. Are you saying that localtime() supports timestamp after the year 2038 on 64-bit AIX? Did you test that time.localtime(2**32) actually works as expected?

I think it worked as expected before 3.8 on 64 bit.

On AIX 64bit, with Python3.7.4,

$ python3_64
Python 3.7.4 (default, Jan 15 2020, 15:46:22)
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
time.struct_time(tm_year=2106, tm_mon=2, tm_mday=7, tm_hour=7, tm_min=28, tm_sec=16, tm_wday=6, tm_yday=38, tm_isdst=0)

and on 32bit

$ python3_32 
Python 3.7.4 (default, Jan 15 2020, 15:50:53) 
[GCC 8.3.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

With the new Python 3.8.1, on both 32 and 64bits,

Python 3.8.1 (default, Jan 30 2020, 11:23:14) 
[GCC 8.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import time; time.localtime(2**32)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: localtime argument out of range