Cpu memory optimization #3602
Cpu memory optimization #3602
Conversation
github-actions
bot
added
component: lowering
cehongwang
force-pushed
the
cpu-memory-optimization
branch
from
a609cfd to
85107c3
Compare
github-actions
bot
removed
the
component: converters
cehongwang
force-pushed
the
cpu-memory-optimization
branch
from
85107c3 to
53c1bcb
Compare
cehongwang
force-pushed
the
cpu-memory-optimization
branch
from
f147421 to
711446c
Compare
github-actions
bot
added
component: tests
|
|
||
| class TRTInterpreterResult(NamedTuple): | ||
| serialized_engine: bytes | ||
| engine: trt.ICudaEngine | bytes |
There was a problem hiding this comment.
@narendasan Should we set it to be engine when it is python runtime and serialized engine if it is cpp runtime? In this way we can do serialization in Interpreter.
There was a problem hiding this comment.
It can just be trt.ICudaEngineand post processing for the cpp runtime
There was a problem hiding this comment.
Like I would have the CPP runtime determine if the engine is live or serialized, and if its live serialize the engine.
github-actions
bot
removed
component: tests
|
|
||
| return rt_cls( | ||
| serialized_engine=interpreter_result.serialized_engine, | ||
| return PythonTorchTensorRTModule( |
There was a problem hiding this comment.
This should be the rt_class still but there should be some preprocessing modifying the arguments
|
|
||
| class TRTInterpreterResult(NamedTuple): | ||
| serialized_engine: bytes | ||
| engine: trt.ICudaEngine | bytes |
There was a problem hiding this comment.
It can just be trt.ICudaEngineand post processing for the cpp runtime
|
|
||
| class TRTInterpreterResult(NamedTuple): | ||
| serialized_engine: bytes | ||
| engine: trt.ICudaEngine | bytes |
There was a problem hiding this comment.
Like I would have the CPP runtime determine if the engine is live or serialized, and if its live serialize the engine.
| @needs_refit # type: ignore[misc] | ||
| def _insert_engine_to_cache(self, hash_val: str, serialized_engine: bytes) -> None: | ||
| def _insert_engine_to_cache(self, hash_val: str, engine: bytes) -> None: | ||
| serialized_engine = engine.serialize() |
There was a problem hiding this comment.
If we are doing this don't we end up paying the serialization cost again?
| self.ctx.requires_output_allocator, | ||
| ) | ||
| else: | ||
| serialized_engine = cuda_engine.serialize() |
There was a problem hiding this comment.
Just have the TRTInterpreter return a live engine in all cases
| def __init__( | ||
| self, | ||
| serialized_engine: Optional[bytes] = None, | ||
| cuda_engine: trt.ICudaEngine = None, |
There was a problem hiding this comment.
The API should support both since uses may use this runtime independently, its just now we either support cuda_engine or serialized_engine
There was a problem hiding this comment.
Same thing for the CPP runtime
Description
Delete the extra copy INetworkDefinition creates
Fixes # (issue)
Type of change
Please delete options that are not relevant and/or add your own.
Checklist: