TokenEstimator

Package: flyte.extras

Protocol for records that can estimate their own token count.

Implement this on your record type and the TokenBatcher will call it automatically when no explicit estimated_tokens is passed to TokenBatcher.submit.

@dataclass
class Prompt:
    text: str

    def estimate_tokens(self) -> int:
        return len(self.text) // 4 + 1

protocol TokenEstimator()

Methods

estimate_tokens()

def estimate_tokens()