Struct SentenceSegmenter 

pub struct SentenceSegmenter { /* private fields */ }

Supports loading sentence break data, and creating sentence break iterators for different string encodings.

Examples

Segment a string:

use icu::segmenter::SentenceSegmenter;
let segmenter = SentenceSegmenter::new();

let breakpoints: Vec<usize> =
    segmenter.segment_str("Hello World").collect();
assert_eq!(&breakpoints, &[0, 11]);

Segment a Latin1 byte string:

use icu::segmenter::SentenceSegmenter;
let segmenter = SentenceSegmenter::new();

let breakpoints: Vec<usize> =
    segmenter.segment_latin1(b"Hello World").collect();
assert_eq!(&breakpoints, &[0, 11]);

Successive boundaries can be used to retrieve the sentences. In particular, the first boundary is always 0, and the last one is the length of the segmented text in code units.

use itertools::Itertools;
let text = "Ceci tuera cela. Le livre tuera l’édifice.";
let sentences: Vec<&str> = segmenter
    .segment_str(text)
    .tuple_windows()
    .map(|(i, j)| &text[i..j])
    .collect();
assert_eq!(
    &sentences,
    &["Ceci tuera cela. ", "Le livre tuera l’édifice."]
);

Implementations

impl SentenceSegmenter

pub fn new() -> Self

Constructs a SentenceSegmenter with an invariant locale and compiled data.

✨ Enabled with the compiled_data Cargo feature.

📚 Help choosing a constructor

pub fn try_new_with_any_provider( provider: &(impl AnyProvider + ?Sized), ) -> Result<Self, SegmenterError>

A version of [Self :: new] that uses custom data provided by an AnyProvider.

📚 Help choosing a constructor

pub fn try_new_unstable<D>(provider: &D) -> Result<Self, SegmenterError>

where D: DataProvider<SentenceBreakDataV1Marker> + ?Sized,

A version of Self::new that uses custom data provided by a DataProvider.

📚 Help choosing a constructor

⚠️ The bounds on provider may change over time, including in SemVer minor releases.

pub fn segment_str<'l, 's>( &'l self, input: &'s str, ) -> SentenceBreakIteratorUtf8<'l, 's>

Creates a sentence break iterator for an str (a UTF-8 string).

There are always breakpoints at 0 and the string length, or only at 0 for the empty string.

pub fn segment_utf8<'l, 's>( &'l self, input: &'s [u8], ) -> SentenceBreakIteratorPotentiallyIllFormedUtf8<'l, 's>

Creates a sentence break iterator for a potentially ill-formed UTF8 string

Invalid characters are treated as REPLACEMENT CHARACTER

There are always breakpoints at 0 and the string length, or only at 0 for the empty string.

pub fn segment_latin1<'l, 's>( &'l self, input: &'s [u8], ) -> SentenceBreakIteratorLatin1<'l, 's>

Creates a sentence break iterator for a Latin-1 (8-bit) string.

There are always breakpoints at 0 and the string length, or only at 0 for the empty string.

pub fn segment_utf16<'l, 's>( &'l self, input: &'s [u16], ) -> SentenceBreakIteratorUtf16<'l, 's>

Creates a sentence break iterator for a UTF-16 string.

There are always breakpoints at 0 and the string length, or only at 0 for the empty string.

Trait Implementations

impl Debug for SentenceSegmenter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SentenceSegmenter

fn default() -> Self

Returns the “default value” for a type.

Layout

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.

Size: 136 bytes