Converting Between datetime, Timestamp, and datetime64 in Python

Internal Representation of numpy.datetime64

The numpy.datetime64 object stores time as integer ticks since the Unix epoch (1970-01-01 00:00:00 UTC), with precision controlled by a parameter ranging from 0 to 9, corresponding to seconds to nanoseconds. For instance, a precision of 3 indicates milliseconds. This design enables efficient storage, but conversions require attention to behavioral differences across numpy versions.

Converting to datetime.datetime

Multiple methods exist for converting numpy.datetime64 to datetime.datetime. In numpy 1.8 and later, using astype(datetime) may return an integer instead of a datetime object, so manual calculation is recommended. First, determine the precision of datetime64; for example, with nanosecond precision, convert ticks to seconds. Code example:

import numpy as np
from datetime import datetime

# Example: Convert datetime64 to datetime
dt64 = np.datetime64('2012-05-01T01:00:00.000000000')
# Calculate seconds since epoch
ns_per_second = 1e9
seconds_since_epoch = dt64.astype('int64') / ns_per_second
dt = datetime.utcfromtimestamp(seconds_since_epoch)
print(dt)  # Output: 2012-05-01 01:00:00

For lower precision datetime64, specify units to simplify. For example, if using second precision, convert directly:

dt64_seconds = np.datetime64('2012-05-01T01:00:00', 's')
dt = datetime.utcfromtimestamp(dt64_seconds.astype('int64'))
print(dt)  # Output: 2012-05-01 01:00:00

In numpy 1.6, astype(datetime) might return a datetime object directly, but it is advisable to check versions for consistency.

Converting to pandas Timestamp

The pandas library offers convenient methods to convert numpy.datetime64 to Timestamp. Use the pd.Timestamp constructor or pd.to_datetime function, with the latter being more flexible and supporting time zones. Code example:

import pandas as pd
import numpy as np

# Direct use of Timestamp constructor
dt64 = np.datetime64('2012-05-01T01:00:00.000000')
ts = pd.Timestamp(dt64)
print(ts)  # Output: 2012-05-01 01:00:00

# Using to_datetime for strings or datetime64
ts_alt = pd.to_datetime(dt64)
print(ts_alt)  # Output: 2012-05-01 01:00:00

For datetime64 with time zones, to_datetime can parse and retain the information. Example:

dt64_tz = np.datetime64('2012-05-01T01:00:00.000000+0100')
ts_tz = pd.to_datetime(dt64_tz)
print(ts_tz)  # Output: 2012-05-01 01:00:00+01:00

Timestamp objects can easily convert to datetime or back using to_pydatetime and to_datetime64 methods.

Handling Time Zones and Precision

Time zone handling is critical in conversions. numpy.datetime64 defaults to UTC, but parsed strings may include offsets. The pandas to_datetime function handles various formats and supports time zone conversion. Code example:

# Localizing time zones
dt64 = np.datetime64('2012-05-01T01:00:00.000000+0100')
ts = pd.to_datetime(dt64)
# Convert to another time zone
ts_utc = ts.tz_convert('UTC')
print(ts_utc)  # Output: 2012-05-01 00:00:00+00:00

Precision issues require attention; high-precision datetime64 may lose information during conversion. For example, nanosecond precision needs scaling when converting to seconds. Use numpy's timedelta64 for assistance:

dt64_ns = np.datetime64('2012-05-01T01:00:00.123456789', 'ns')
# Convert to second precision
dt64_s = dt64_ns.astype('datetime64[s]')
print(dt64_s)  # Output: 2012-05-01T01:00:00

Regarding version compatibility, astype behavior changed after numpy 1.8, so using unified methods is recommended to avoid errors.

Code Examples and Best Practices

A comprehensive example demonstrates the full conversion process. First, create objects of each type, then perform bidirectional conversions:

import datetime
import numpy as np
import pandas as pd

# Create sample objects
dt = datetime.datetime(2012, 5, 1)
dt64 = np.datetime64(dt)
ts = pd.Timestamp(dt)

# datetime64 to datetime
# Method 1: Using calculation
seconds = (dt64 - np.datetime64('1970-01-01T00:00:00Z')) / np.timedelta64(1, 's')
dt_from_dt64 = datetime.datetime.utcfromtimestamp(seconds)
print(f"datetime from datetime64: {dt_from_dt64}")

# Method 2: Using pandas
ts_from_dt64 = pd.Timestamp(dt64)
dt_from_ts = ts_from_dt64.to_pydatetime()
print(f"datetime from Timestamp: {dt_from_ts}")

# Reverse conversion: datetime to datetime64
dt64_from_dt = np.datetime64(dt)
print(f"datetime64 from datetime: {dt64_from_dt}")

# Handle edge cases, such as long integer values
dt64_long = np.datetime64('2002-06-28T01:00:00.000000000+0100')
# In numpy 1.8, it may return a long, requiring manual handling
if isinstance(dt64_long.astype(object), (int, long)):
    ns = 1e-9
    dt_corrected = datetime.utcfromtimestamp(dt64_long.astype('int64') * ns)
    print(f"Corrected datetime: {dt_corrected}")

Best practices include prioritizing pandas functions for complex time zones, checking numpy versions, and validating data types before conversion. Avoid relying solely on astype due to potential inconsistencies.

Conclusion

Converting between datetime, Timestamp, and datetime64 requires understanding their internal mechanisms. numpy.datetime64 stores time as ticks, while pandas Timestamp offers rich functionality. Efficient conversion is achievable through manual calculation or pandas tools. Time zone and precision handling are key; using unified methods in projects ensures compatibility. The approaches in this article apply to most scenarios, but note that library updates may introduce changes.