PHP  
downloads | documentation | faq | getting help | mailing lists | reporting bugs | php.net sites | links | my php.net 
search for in the  
<strstrstrtolower>
view the version of this page
Last updated: Thu, 21 Aug 2003

strtok

(PHP 3, PHP 4 )

strtok -- Tokenize string

Description

string strtok ( string arg1, string arg2)

strtok() splits a string (arg1) into smaller strings (tokens), with each token being delimited by any character from arg2. That is, if you have a string like "This is an example string" you could tokenize this string into its individual words by using the space character as the token.

Example 1. strtok() example

<?php
$string = "This is\tan example\nstring";
/* Use tab and newline as tokenizing characters as well  */
$tok = strtok($string," \n\t");
while ($tok) {
    echo "Word=$tok<br>";
    $tok = strtok(" \n\t");
}
?>

Note that only the first call to strtok uses the string argument. Every subsequent call to strtok only needs the token to use, as it keeps track of where it is in the current string. To start over, or to tokenize a new string you simply call strtok with the string argument again to initialize it. Note that you may put multiple tokens in the token parameter. The string will be tokenized when any one of the characters in the argument are found.

The behavior when an empty part was found changed with PHP 4.1.0. The old behavior returned an empty string, while the new, correct, behavior simply skips the part of the string:

Example 2. Old strtok() behavior

<?php
$first_token  = strtok('/something', '/');
$second_token = strtok('/');
var_dump ($first_token, $second_token);

/* Output:
    string(0) ""
    string(9) "something"
*/    
?>

Example 3. New strtok() behavior

<?php
$first_token  = strtok('/something', '/');
$second_token = strtok('/');
var_dump ($first_token, $second_token);

/* Output:
    string(9) "something"
    bool(false)
*/    
?>

Also be careful that your tokens may be equal to "0". This evaluates to FALSE in conditional expressions.

See also split() and explode().



add a note add a note User Contributed Notes
strtok
jrust AT rustyparts DOT com
24-May-2003 09:09
Had a website which was using way too many of the old functionality of strtok to convert to the new >PHP 4.1.0 way so I wrote this function to mimic the way strtok was done prior to 4.1.0

function strtok_old($string, $delim = null) {
    static $origDelim, $origString, $origPos;
    if (!isset($origDelim)) {
        $origDelim = null;
    }
    if (!isset($origString)) {
        $origString = null;
    }
    if (!isset($origPos)) {
        $origPos = null;
    }

    // continuing an already started strtok
    if ($string == $origDelim) {
        $string = $origString;
        $delim = $origDelim;
    }
    // else starting from scratch
    else {
        $origString = $string;
        $origDelim = $delim;
        $origPos = 0;
    }

    if ($origPos !== false && $origPos < strlen($string)) {
        $newPos = strpos($string, $delim, $origPos);
    }
    else {
        $newPos = false;
    }

    // the token wasn't found, go to end of string
    if ($newPos === false) {
        $newPos = strlen($string);
    }

    $return = substr($string, $origPos, ($newPos - $origPos));
    $origPos = ++$newPos;
    return $return;
}
shaun at phplabs dot com
05-Dec-2002 01:57
Here's some code to extract the first part of a long paragraph, e.g. to use as a summary. Starting at the beginning of the paragraph it gets as many complete sentences as are necessary to contain $limit words. For example, with $limit at 20 it would return the first two sentences of the paragraph you're reading right now (the first 20 words plus the rest of the sentence in which the limit was hit).

function summarize($paragraph, $limit){
  $tok = strtok($paragraph, " ");
  while($tok){
    $text .= " $tok";
    $words++;
    if(($words >= $limit) && ((substr($tok, -1) == "!")||(substr($tok, -1) == ".")))
      break;
    $tok = strtok(" ");
  }
return ltrim($text);
}

Might be a better way to do this, but it worked for me. Hope you find it useful!
30-Aug-2002 03:58
once again it is NOT strtok that fails but your (dxue69@hotmail.com) loop test.
That the string "0" evaluates to false in a boolean context is emphasized in the last sentence of this manual page!!!

if you have php4 use while($str !== false)
if you have php3 you can use while(is_string($str)).

my humble piece of advice to the documentation editors if they want to avoid editing this same mistake over and over again (not only in this page but in the whole manual, this is probably the most frequent error in posts throughout the website!) would be to use correct comparison tests in the exemples

Ivan
dxue69 at hotmail dot com
26-Jun-2002 06:25
If you tokenize a string containing "0" such as "0,1,2"
strtok will fail when tokenizing the 0 in the beggining. 
$tok=strtok($begin_with_0, ",")
while($tok){
  echo("$tok ");
}
will not print anything.  To get around this you need to use split()
$tok_array=split(",", $begin_with_0);
then 0,1,2 will be in $tok_array
desolate19 at NOSPAM dot hotmail dot com
03-May-2002 05:28
Here is yet another explanation of strtok for the explode/split comments.

You can do things with strtok that you can't do with explode/split. explode breaks a string using another string, split breaks a string using a regular expression.  strtok breaks a string using single _characters_ , but the best part is you can use multiple characters at the same time.

For example, if you are accepting user input and aren't sure how the user will decide to divide up their data you could choose to tokenize on spaces, hyphens, slashes and backslashes ALL AT THE SAME TIME:

<?PHP

$teststr 
"blah1 blah2/blah3-blah4\\blah5";

$tok strtok($teststr," /-\\");
while (
$tok !== FALSE)
{
 
$toks[] = $tok;
 
$tok strtok(" /-\\");
}

while (list(
$k,$v) = each($toks))
{
  print (
"$k => $v&lt;BR&gt;\n");
}

?>

/* OUTPUT:
0 => blah1
1 => blah2
2 => blah3
3 => blah4
4 => blah5
*/

You can't do that with explode, and this should be faster than using split because split uses regular expressions.

And for the comments about explode/split putting your output into an array... as you can see, it's not hard to work with arrays in PHP.
alkiera at ihatespam dot hotmail dot com
22-Mar-2002 04:36
In addition to the difference mckay pointed out above, which is apparently fixed in php 4.1+, is another difference between the php and C versions of strtok().

Unlike the C version, php's strtok() does not seem to eat the tokens, they get left in the stream, to be returned on the next call.  Seems very odd, though it might be a design decision, it doesn't say.
geigh at wad dot com
02-Mar-2002 08:52
here is a useful example.  It will take a comma deliminated list and print each phrase from it.

$bruno="lame,a piece of crap,retarded,pathetic,a one button wonder";

$macOS=strtok($bruno,",");
while ($macOS)
{
  echo "MacOS is $macOS\n";
  $macOS=strtok($macOS,",");
}
12-Dec-2001 04:57
The example is unnecessarily confusing for beginners.

1) It is NOT strtok that fails when the returned string evaluates
to false in conditional expression, it is the loop test. A correct test is
while($tok !== false)

2) the same functionality (AS THE EXAMPLE) can be obtained with
explode. Note that if you only need the first few tokens you can
put a limit on explode!! read the manual :)
array explode (string separator, string string [, INT LIMIT])

What you can NOT do with explode (or split) is changing the
separator after a token is returned, as for example, when parsing a
string along a simple format :

$styleStr = "color:#FFFFFF;font-size:12";
$key = strtok($styleStr,":");
while ($key !== false){
 $styleTab[$key]= strtok(";"); // do not change the target
 $key = strtok(":"); // string, just the separator list
}

$styleTab is array("color"=>"#FFFFFF","font-size"=>"12")

If you need the remaining of the string do :
$remaining = strtok(''); //(empty separator)

Ivan
rbronosky at mac dot com
01-Jun-2001 12:43
The reason for using strtok() is to avoid needlessly duplicating data.  If you have a string with the complete data in it, then why duplicate it into an array.  This takes time and system resources.  I know it does not seem like a big deal.  But with lots of traffic, or with a large string (ie.  $str=fread($A_500k_File); ) the efficiency is extremely important.

Efficiency being something that MS Windows users are not accustomed to anyway.

-- for REAL power develop using MacOS, BBEdit, and PHP. The "Holy Trinity"!

-Bruno
rbronosky at mac dot com
31-May-2001 11:27
The reason for using strtok() is to avoid needlessly duplicating data.  If you have a string with the complete data in it, then why duplicate it into an array.  This takes time and system resources.  I know it does not seem like a big deal.  But with lots of traffic, or with a large string (ie.  $str=fread($A_500k_File); ) the efficiency is extremely important.

Efficiency being something that MS Windows users are not accustomed to anyway.

-- for REAL power develop using MacOS, BBEdit, and PHP. The "Holy Trinity"!

-Bruno
slt at municipium dot com dot pl
15-Feb-2001 03:12
As 'mckay' wrote, strtok 2nd argument is a list of tokens, not a string delimiter. It's not so obvious as one may think and it may be confusing for beginners like me. So, in the docs, it should state sth. like that
strtok(string where2search, char token2cut).
And for the above split-lover =) 'tysonlt' -> it's better to use explode bcoz it's lighter than split (to quote original manual: "(...) use explode(), which doesn't incur the overhead of the regular expression engine")
regards,
     StarLight
mxq at supergamez dot hu
08-Jan-2001 03:55
strtok () cannot always be fully replaced by explode. You may only need the first several tokens from a long string, then do not use explode to explode the whole string - you just waste time.

strtok () seemed not to work when I tried to put the result into the same variable as I used for arg1 (for e.g., $strx = strtok ($strx, "."); simply made $strx be empty). This must be something with the pointer used. Use of an auxiliary string is needed so something like this $aux = strtok ($strx, "."); $strx = $aux;
tysonlt at spamless dot webmedia dot com dot au
04-Oct-2000 06:34
Why use strtok at all?

If it's so flaky, why not just use split?

eg.
$token_array = split("$delim", $string);

Then you can use all the nice array functions on it! :)
David dot Mazur at student dot kuleuven dot ac dot be
16-Aug-2000 06:40
If you want to tokenize only part of the string, and store the "untokenized" part in some
variable, you have to call strtok one last time with separator "" (i.e. the empty string).
vksgeneric at hotmail dot com
13-Jul-2000 05:11
a lot of people implied it by now, but nobody wrote the magic solution. If you have, for example, a couple delimiters following each other, the example in this chapter:

$tok = strtok ($string," ");
  while ($tok) {
...

will fail. Instead, try something like this:

$tok = strtok ($string, " ");
  while(!($tok === FALSE)){
...

(Yes, this is three '=' in a row...)
domidahl at gmx dot net
10-May-2000 11:08
If you tokenize a string containing "0" use strlen($tok) as breaking-condition of the loop.
This works because strval(false) is "".
mckay at et dot byu dot edu
04-May-2000 09:43
Note that php's strtok() is NOT the same as the C library function strtok().  From the documentation on the C library function:
    strtok(s1,s2) considers the string s1 to consist of a sequence of
    zero or more text tokens separated by spans of one or more
    characters from the separator string s2.

Replace "one or more" above with "exactly one" and you'd have a description of what php's strtok() does.  Thus, to php's strtok, "three spaces   here" maps to the tokens  "three" "spaces" "" "" "here".  The C library function (much more useful) would map the same string to the tokens "three" "spaces" "here".

I'm using php version 3.1.12--perhaps this will be changed in later versions
ridcully at gmx dot li
07-Mar-2000 08:51
As mentioned in the documentation, strtok() takes the string to be tokenized only when called the first time. It stores a pointer to the string internally and subsequent calls to strtok() (without the string parameter) use that pointer.
kgrose at web dot net
03-Dec-1999 09:53
It's not the newline that causes strtok to stop working, it's if it runs into whatever character you've specified as the delimiter several times in a row (like more than 2 spaces if space is your delimiter).
abhijit at mds dot rmit dot edu dot au
01-Dec-1999 09:46
If you want to split a string using a "string" as the delimiter- try explode. Works great!

<strstrstrtolower>
 Last updated: Thu, 21 Aug 2003
show source | credits | sitemap | mirror sites 
Copyright © 2001-2003 The PHP Group
All rights reserved.
This mirror generously provided by: http://php.mirrors.ilisys.com.au/
Last updated: Sat 01 Nov 2003 04:13:36 EST EST