/*
 * Copyright © 2009 CNRS, INRIA, Université Bordeaux 1
 * Copyright © 2009-2010 Cisco Systems, Inc.  All rights reserved.
 * See COPYING in top-level directory.
 */

/*! \mainpage Hardware Locality

<h1 class="sub">Portable abstraction of hierarchical architectures for high-performance computing</h1>

<hr>

\htmlonly
<div class="section" id="introduction">
\endhtmlonly
\section Introduction

hwloc provides command line tools and a C API to obtain the
hierarchical map of key computing elements, such as: NUMA memory
nodes, shared caches, processor sockets, processor cores, and
processing units (logical processors or "threads").
hwloc also gathers various attributes such as
cache and memory information, and is portable across a variety of
different operating systems and platforms.

hwloc primarily aims at helping high-performance computing (HPC)
applications, but is also applicable to any project seeking to exploit
code and/or data locality on modern computing platforms.

*** Note that the hwloc project represents the merger of the
libtopology project from INRIA and the Portable Linux Processor
Affinity (PLPA) sub-project from Open MPI.  <em>Both of these prior
projects are now deprecated.</em> The first hwloc release is essentially a
"re-branding" of the libtopology code base, but with both a few
genuinely new features and a few PLPA-like features added in.  More
new features and more PLPA-like features will be added to hwloc over
time. See \ref switchfromplpa for more details about converting
your application from PLPA to hwloc.

hwloc supports the following operating systems:

<ul>
<li>Linux (including old kernels not having sysfs topology
information, with knowledge of cpusets, offline cpus, ScaleMP vSMP,
and Kerrighed support)</li>
<li>Solaris</li>
<li>AIX</li>
<li>Darwin / OS X</li>
<li>FreeBSD and its variants, such as kFreeBSD/GNU</li>
<li>OSF/1 (a.k.a., Tru64)</li>
<li>HP-UX</li>
<li>Microsoft Windows</li>
</ul>

hwloc only reports the number of processors on unsupported operating
systems; no topology information is available.

For development and debugging purposes, hwloc also offers the ability to
work on "fake" topologies:

<ul>
  <li> Symmetrical tree of resources generated from a list of level arities</li>
  <li> Remote machine simulation through the gathering of Linux sysfs topology files</li>
</ul>

hwloc can display the topology in a human-readable format, either in
graphical mode (X11), or by exporting in one of several different
formats, including: plain text, PDF, PNG, and FIG (see \ref cli_examples
below).  Note that some of the export formats require additional
support libraries.

hwloc offers a programming interface for manipulating topologies and
objects. It also brings a powerful CPU bitmap API that is used to
describe topology objects location on physical/logical processors. See
the \ref interface below. It may also be used to binding applications
onto certain cores or memory nodes. Several utility programs are also
provided to ease command-line manipulation of topology objects,
binding of processes, and so on.

\htmlonly
</div><div class="section" id="installation">
\endhtmlonly
\section installation Installation

hwloc (http://www.open-mpi.org/projects/hwloc/) is available under the
BSD license.  It is hosted as a sub-project of the overall Open MPI
project (http://www.open-mpi.org/).  Note that hwloc does not require
any functionality from Open MPI -- it is a wholly separate (and much
smaller!) project and code base.  It just happens to be hosted as part
of the overall Open MPI project.

Nightly development snapshots are available on the web site.
Additionally, the code can be directly checked out of Subversion:

\code
shell$ svn checkout http://svn.open-mpi.org/svn/hwloc/trunk hwloc-trunk
shell$ cd hwloc-trunk
shell$ ./autogen.sh
\endcode

Note that GNU Autoconf >=2.63, Automake >=1.10 and Libtool >=2.2.6 are
required when building from a Subversion checkout.

Installation by itself is the fairly common GNU-based process:

\code
shell$ ./configure --prefix=...
shell$ make
shell$ make install
\endcode

The hwloc command-line tool "lstopo" produces human-readable topology
maps, as mentioned above.  It can also export maps to the "fig" file
format.  Support for PDF, Postscript, and PNG exporting is provided if
the "Cairo" development package can be found when hwloc is configured
and build.  Similarly, lstopo's XML support requires the libxml2
development package.

\htmlonly
</div><div class="section" id="cli_examples">
\endhtmlonly
\section cli_examples CLI Examples

On a 4-socket 2-core machine with hyperthreading, the \c lstopo tool
may show the following graphic output:

\image html dudley.png
\image latex dudley.png "" width=9cm

Here's the equivalent output in textual form:

\verbatim
Machine (16GB)
  Socket #0 + L3 #0 (4096KB)
    L2 #0 (1024KB) + L1 #0 (16KB) + Core #0
      PU #0 (phys=0)
      PU #1 (phys=8)
    L2 #1 (1024KB) + L1 #1 (16KB) + Core #1
      PU #2 (phys=4)
      PU #3 (phys=12)
  Socket #1 + L3 #1 (4096KB)
    L2 #2 (1024KB) + L1 #2 (16KB) + Core #2
      PU #4 (phys=1)
      PU #5 (phys=9)
    L2 #3 (1024KB) + L1 #3 (16KB) + Core #3
      PU #6 (phys=5)
      PU #7 (phys=13)
  Socket #2 + L3 #2 (4096KB)
    L2 #4 (1024KB) + L1 #4 (16KB) + Core #4
      PU #8 (phys=2)
      PU #9 (phys=10)
    L2 #5 (1024KB) + L1 #5 (16KB) + Core #5
      PU #10 (phys=6)
      PU #11 (phys=14)
  Socket #3 + L3 #3 (4096KB)
    L2 #6 (1024KB) + L1 #6 (16KB) + Core #6
      PU #12 (phys=3)
      PU #13 (phys=11)
    L2 #7 (1024KB) + L1 #7 (16KB) + Core #7
      PU #14 (phys=7)
      PU #15 (phys=15)
\endverbatim

Finally, here's the equivalent output in XML.  Long lines were
artificially broken for document clarity (in the real output, each XML
tag is on a single line), and only socket #0 is shown for brevity:

\verbatim
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc.dtd">
<topology>
  <object type="Machine" os_level="-1" os_index="0" cpuset="0x0000ffff" 
      complete_cpuset="0x0000ffff" online_cpuset="0x0000ffff" 
      allowed_cpuset="0x0000ffff" 
      dmi_board_vendor="Dell Computer Corporation" dmi_board_name="0RD318" 
      local_memory="16648183808">
    <page_type size="4096" count="4064498"/>
    <page_type size="2097152" count="0"/>
    <object type="Socket" os_level="-1" os_index="0" cpuset="0x00001111" 
        complete_cpuset="0x00001111" online_cpuset="0x00001111" 
        allowed_cpuset="0x00001111">
      <object type="Cache" os_level="-1" cpuset="0x00001111" 
          complete_cpuset="0x00001111" online_cpuset="0x00001111" 
          allowed_cpuset="0x00001111" cache_size="4194304" depth="3" 
          cache_linesize="64">
        <object type="Cache" os_level="-1" cpuset="0x00000101" 
            complete_cpuset="0x00000101" online_cpuset="0x00000101" 
            allowed_cpuset="0x00000101" cache_size="1048576" depth="2" 
            cache_linesize="64">
          <object type="Cache" os_level="-1" cpuset="0x00000101" 
              complete_cpuset="0x00000101" online_cpuset="0x00000101" 
              allowed_cpuset="0x00000101" cache_size="16384" depth="1" 
              cache_linesize="64">
            <object type="Core" os_level="-1" os_index="0" cpuset="0x00000101" 
                complete_cpuset="0x00000101" online_cpuset="0x00000101" 
                allowed_cpuset="0x00000101">
              <object type="PU" os_level="-1" os_index="0" cpuset="0x00000001" 
                  complete_cpuset="0x00000001" online_cpuset="0x00000001" 
                  allowed_cpuset="0x00000001"/>
              <object type="PU" os_level="-1" os_index="8" cpuset="0x00000100" 
                  complete_cpuset="0x00000100" online_cpuset="0x00000100" 
                  allowed_cpuset="0x00000100"/>
            </object>
          </object>
        </object>
        <object type="Cache" os_level="-1" cpuset="0x00001010" 
            complete_cpuset="0x00001010" online_cpuset="0x00001010" 
            allowed_cpuset="0x00001010" cache_size="1048576" depth="2" 
            cache_linesize="64">
          <object type="Cache" os_level="-1" cpuset="0x00001010" 
              complete_cpuset="0x00001010" online_cpuset="0x00001010" 
              allowed_cpuset="0x00001010" cache_size="16384" depth="1" 
              cache_linesize="64">
            <object type="Core" os_level="-1" os_index="1" cpuset="0x00001010" 
                complete_cpuset="0x00001010" online_cpuset="0x00001010" 
                allowed_cpuset="0x00001010">
              <object type="PU" os_level="-1" os_index="4" cpuset="0x00000010" 
                  complete_cpuset="0x00000010" online_cpuset="0x00000010" 
                  allowed_cpuset="0x00000010"/>
              <object type="PU" os_level="-1" os_index="12" cpuset="0x00001000" 
                  complete_cpuset="0x00001000" online_cpuset="0x00001000" 
                  allowed_cpuset="0x00001000"/>
            </object>
          </object>
        </object>
      </object>
    </object>
    <!-- ...other sockets listed here ... -->
  </object>
</topology>
\endverbatim

On a 4-socket 2-core Opteron NUMA machine, the \c lstopo tool may show
the following graphic output:

\image html hagrid.png
\image latex hagrid.png width=\textwidth

Here's the equivalent output in textual form:

\verbatim
Machine (64GB)
  NUMANode #0 (phys=0 8190MB) + Socket #0
    L2 #0 (1024KB) + L1 #0 (64KB) + Core #0 + PU #0 (phys=0)
    L2 #1 (1024KB) + L1 #1 (64KB) + Core #1 + PU #1 (phys=1)
  NUMANode #1 (phys=1 8192MB) + Socket #1
    L2 #2 (1024KB) + L1 #2 (64KB) + Core #2 + PU #2 (phys=2)
    L2 #3 (1024KB) + L1 #3 (64KB) + Core #3 + PU #3 (phys=3)
  NUMANode #2 (phys=2 8192MB) + Socket #2
    L2 #4 (1024KB) + L1 #4 (64KB) + Core #4 + PU #4 (phys=4)
    L2 #5 (1024KB) + L1 #5 (64KB) + Core #5 + PU #5 (phys=5)
  NUMANode #3 (phys=3 8192MB) + Socket #3
    L2 #6 (1024KB) + L1 #6 (64KB) + Core #6 + PU #6 (phys=6)
    L2 #7 (1024KB) + L1 #7 (64KB) + Core #7 + PU #7 (phys=7)
  NUMANode #4 (phys=4 8192MB) + Socket #4
    L2 #8 (1024KB) + L1 #8 (64KB) + Core #8 + PU #8 (phys=8)
    L2 #9 (1024KB) + L1 #9 (64KB) + Core #9 + PU #9 (phys=9)
  NUMANode #5 (phys=5 8192MB) + Socket #5
    L2 #10 (1024KB) + L1 #10 (64KB) + Core #10 + PU #10 (phys=10)
    L2 #11 (1024KB) + L1 #11 (64KB) + Core #11 + PU #11 (phys=11)
  NUMANode #6 (phys=6 8192MB) + Socket #6
    L2 #12 (1024KB) + L1 #12 (64KB) + Core #12 + PU #12 (phys=12)
    L2 #13 (1024KB) + L1 #13 (64KB) + Core #13 + PU #13 (phys=13)
  NUMANode #7 (phys=7 8192MB) + Socket #7
    L2 #14 (1024KB) + L1 #14 (64KB) + Core #14 + PU #14 (phys=14)
    L2 #15 (1024KB) + L1 #15 (64KB) + Core #15 + PU #15 (phys=15)
\endverbatim

And here's the equivalent output in XML.  Similar to above, line
breaks were added and only PU#0 is shown for brevity:

\verbatim
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc.dtd">
<topology>
  <object type="Machine" os_level="-1" os_index="0" cpuset="0x000000ff" 
      complete_cpuset="0x000000ff" online_cpuset="0x000000ff" 
      allowed_cpuset="0x000000ff" nodeset="0x000000ff" 
      complete_nodeset="0x000000ff" allowed_nodeset="0x000000ff" 
      dmi_board_vendor="TYAN Computer Corp" dmi_board_name="S4881 ">
    <page_type size="4096" count="0"/>
    <page_type size="2097152" count="0"/>
    <object type="NUMANode" os_level="-1" os_index="0" cpuset="0x00000003" 
        complete_cpuset="0x00000003" online_cpuset="0x00000003" 
        allowed_cpuset="0x00000003" nodeset="0x00000001" 
        complete_nodeset="0x00000001" allowed_nodeset="0x00000001" 
        local_memory="7514177536">
      <page_type size="4096" count="1834516"/>
      <page_type size="2097152" count="0"/>
      <object type="Socket" os_level="-1" os_index="0" cpuset="0x00000003" 
          complete_cpuset="0x00000003" online_cpuset="0x00000003" 
          allowed_cpuset="0x00000003" nodeset="0x00000001" 
          complete_nodeset="0x00000001" allowed_nodeset="0x00000001">
        <object type="Cache" os_level="-1" cpuset="0x00000001" 
            complete_cpuset="0x00000001" online_cpuset="0x00000001" 
            allowed_cpuset="0x00000001" nodeset="0x00000001" 
            complete_nodeset="0x00000001" allowed_nodeset="0x00000001" 
            cache_size="1048576" depth="2" cache_linesize="64">
          <object type="Cache" os_level="-1" cpuset="0x00000001" 
              complete_cpuset="0x00000001" online_cpuset="0x00000001" 
              allowed_cpuset="0x00000001" nodeset="0x00000001" 
              complete_nodeset="0x00000001" allowed_nodeset="0x00000001" 
              cache_size="65536" depth="1" cache_linesize="64">
            <object type="Core" os_level="-1" os_index="0" 
                cpuset="0x00000001" complete_cpuset="0x00000001" 
                online_cpuset="0x00000001" allowed_cpuset="0x00000001" 
                nodeset="0x00000001" complete_nodeset="0x00000001" 
                allowed_nodeset="0x00000001">
              <object type="PU" os_level="-1" os_index="0" cpuset="0x00000001" 
                  complete_cpuset="0x00000001" online_cpuset="0x00000001" 
                  allowed_cpuset="0x00000001" nodeset="0x00000001" 
                  complete_nodeset="0x00000001" allowed_nodeset="0x00000001"/>
            </object>
          </object>
        </object>
  <!-- ...more objects listed here ... -->
</topology>
\endverbatim

On a 2-socket quad-core Xeon (pre-Nehalem, with 2 dual-core dies into
each socket):

\image html emmett.png
\image latex emmett.png "" width=7cm

Here's the same output in text form:

\verbatim
Machine (16GB)
  Socket #0
    L2 #0 (4096KB)
      L1 #0 (32KB) + Core #0 + PU #0 (phys=0)
      L1 #1 (32KB) + Core #1 + PU #1 (phys=4)
    L2 #1 (4096KB)
      L1 #2 (32KB) + Core #2 + PU #2 (phys=2)
      L1 #3 (32KB) + Core #3 + PU #3 (phys=6)
  Socket #1
    L2 #2 (4096KB)
      L1 #4 (32KB) + Core #4 + PU #4 (phys=1)
      L1 #5 (32KB) + Core #5 + PU #5 (phys=5)
    L2 #3 (4096KB)
      L1 #6 (32KB) + Core #6 + PU #6 (phys=3)
      L1 #7 (32KB) + Core #7 + PU #7 (phys=7)
\endverbatim

And the same output in XML (line breaks added, only PU#0 shown):

\verbatim
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc.dtd">
<topology>
  <object type="Machine" os_level="-1" os_index="0" cpuset="0x000000ff" 
      complete_cpuset="0x000000ff" online_cpuset="0x000000ff" 
      allowed_cpuset="0x000000ff" dmi_board_vendor="Dell Inc." 
      dmi_board_name="0NR282" local_memory="16865292288">
    <page_type size="4096" count="4117503"/>
    <page_type size="2097152" count="0"/>
    <object type="Socket" os_level="-1" os_index="0" cpuset="0x00000055" 
        complete_cpuset="0x00000055" online_cpuset="0x00000055" 
        allowed_cpuset="0x00000055">
      <object type="Cache" os_level="-1" cpuset="0x00000011" 
          complete_cpuset="0x00000011" online_cpuset="0x00000011" 
          allowed_cpuset="0x00000011" cache_size="4194304" depth="2" 
          cache_linesize="64">
        <object type="Cache" os_level="-1" cpuset="0x00000001" 
            complete_cpuset="0x00000001" online_cpuset="0x00000001" 
            allowed_cpuset="0x00000001" cache_size="32768" depth="1" 
            cache_linesize="64">
          <object type="Core" os_level="-1" os_index="0" cpuset="0x00000001" 
              complete_cpuset="0x00000001" online_cpuset="0x00000001" 
              allowed_cpuset="0x00000001">
            <object type="PU" os_level="-1" os_index="0" cpuset="0x00000001" 
                complete_cpuset="0x00000001" online_cpuset="0x00000001" 
                allowed_cpuset="0x00000001"/>
          </object>
        </object>
        <object type="Cache" os_level="-1" cpuset="0x00000010" 
            complete_cpuset="0x00000010" online_cpuset="0x00000010" 
            allowed_cpuset="0x00000010" cache_size="32768" depth="1" 
            cache_linesize="64">
          <object type="Core" os_level="-1" os_index="1" cpuset="0x00000010" 
              complete_cpuset="0x00000010" online_cpuset="0x00000010" 
              allowed_cpuset="0x00000010">
            <object type="PU" os_level="-1" os_index="4" cpuset="0x00000010" 
                complete_cpuset="0x00000010" online_cpuset="0x00000010" 
                allowed_cpuset="0x00000010"/>
          </object>
        </object>
      </object>
  <!-- ...more objects listed here ... -->
</topology>
\endverbatim

\htmlonly
</div><div class="section" id="interface">
\endhtmlonly

\section interface Programming Interface

The basic interface is available in hwloc.h.  It essentially offers
low-level routines for advanced programmers that want to manually
manipulate objects and follow links between them.  Documentation for
everything in hwloc.h are provided later in this document.  Developers
should also look at hwloc/helper.h (and also in this document, which
provides good higher-level topology traversal examples.

To precisely define the vocabulary used by hwloc, a \ref termsanddefs
section is available and should probably be read first.  

Each hwloc object contains a cpuset describing the list of processing
units that it contains.  These cpusets may be used for \ref
hwlocality_binding.  hwloc offers an extensive cpuset manipulation
interface in hwloc/cpuset.h.

Moreover, hwloc also comes with additional helpers for
interoperability with several commonly used environments.
See the \ref interoperability section for details.

The complete API documentation is available in a full set of HTML
pages, man pages, and self-contained PDF files (formatted for both
both US letter and A4 formats) in the source tarball in
doc/doxygen-doc/.  

<strong>NOTE:</strong> If you are building the documentation from a
Subversion checkout, you will need to have Doxygen and pdflatex
installed -- the documentation will be built during the normal "make"
process.  The documentation is installed during "make install" to
$prefix/share/doc/hwloc/ and your systems default man page tree (under
$prefix, of course).

\subsection portability Portability (especially when using the API or XML)

As shown in \ref cli_examples, hwloc can obtain information on a wide
variety of hardware topologies.  However, some platforms and/or
operating system versions will only report a subset of this
information.  For example, on an PPC64-based system with 32 cores
(each with 2 hardware threads) running a default 2.6.18-based kernel
from RHEL 5.4, hwloc is only able to glean information about NUMA
nodes and processor units (PUs).  No information about caches,
sockets, or cores is available.

Here's the graphic output from lstopo on this platform when
Simultaneous Multi-Threading (SMT) is enabled:

\image html ppc64-with-smt.png
\image latex ppc64-with-smt.pdf "" width=\textwidth

And here's the graphic output from lstopo on this platform when SMT is
disabled:

\image html ppc64-without-smt.png
\image latex ppc64-without-smt.pdf "" width=\textwidth

Notice that hwloc only sees half the PUs when SMT is disabled.  PU#15,
for example, seems to change location from NUMA node #0 to #1.  In
reality, no PUs "moved" -- they were simply re-numbered when hwloc
only saw half as many.  Hence, PU#15 in the SMT-disabled picture
probably corresponds to PU#30 in the SMT-enabled picture.  

This same "PUs have disappeared" effect can be seen on other platforms
-- even platforms / OSs that provide much more information than the
above PPC64 system.  This is an unfortunate side-effect of how
operating systems report information to hwloc.

Note that upgrading the Linux kernel on the same PPC64 system
mentioned above to 2.6.34, hwloc is able to discover all the topology
information.  The following picture shows the entire topology layout
when SMT is enabled:

\image html ppc64-full-with-smt.png
\image latex ppc64-full-with-smt.pdf "" width=\textwidth

Developers using the hwloc API or XML output for portable applications
should therefore be extremely careful to not make any assumptions
about the structure of data that is returned.  For example, per the
above reported PPC topology, it is not safe to assume that PUs will
always be descendants of cores.

Additionally, future hardware may insert new topology elements that
are not available in this version of hwloc.  Long-lived applications
that are meant to span multiple different hardware platforms should
also be careful about making structure assumptions.  For example,
there may someday be an element "lower" than a PU, or perhaps a new
element may exist between a core and a PU.

\subsection interface_example API Example

The following small C example (named ``hwloc-hello.c'') prints the
topology of the machine and bring the process to the first logical processor
of the second core of the machine.

\include hwloc-hello.c

hwloc provides a \c pkg-config executable to obtain relevant compiler
and linker flags.  For example, it can be used thusly to compile
applications that utilize the hwloc library (assuming GNU Make):

\verbatim
CFLAGS += $(pkg-config --cflags hwloc)
LDLIBS += $(pkg-config --libs hwloc)
cc hwloc-hello.c $(CFLAGS) -o hwloc-hello $(LDLIBS)
\endverbatim

On a machine with 4GB of RAM and 2 processor sockets -- each socket of
which has two processing cores -- the output from running \c
hwloc-hello could be something like the following:

\verbatim
shell$ ./hwloc-hello
*** Objects at level 0
Index 0: Machine(3938MB)
*** Objects at level 1
Index 0: Socket#0
Index 1: Socket#1
*** Objects at level 2
Index 0: Core#0
Index 1: Core#1
Index 2: Core#3
Index 3: Core#2
*** Objects at level 3
Index 0: PU#0
Index 1: PU#1
Index 2: PU#2
Index 3: PU#3
*** Printing overall tree
Machine(3938MB)
  Socket#0
    Core#0
      PU#0
    Core#1
      PU#1
  Socket#1
    Core#3
      PU#2
    Core#2
      PU#3
*** 2 socket(s)
shell$ 
\endverbatim

\htmlonly
</div><div class="section" id="bugs">
\endhtmlonly
\section bugs Questions and Bugs

Questions should be sent to the devel mailing
list (http://www.open-mpi.org/community/lists/hwloc.php).
Bug reports should be reported in the tracker
(https://svn.open-mpi.org/trac/hwloc/).

If hwloc discovers an incorrect topology for your machine, the very
first thing you should check is to ensure that you have the most
recent updates installed for your operating system.  Indeed, most of
hwloc topology discovery relies on hardware information retrieved
through the operation system (e.g., via the /sys virtual filesystem of
the Linux kernel).  If upgrading your OS or Linux kernel does not
solve your problem, you may also want to ensure that you are running
the most recent version of the BIOS for your machine.

If those things fail, contact us on the mailing list for additional
help.  Please attach the output of lstopo after having given the
--enable-debug option to ./configure and rebuilt completely, to get
debugging output.

\htmlonly
</div><div class="section" id="credits">
\endhtmlonly
\section history History / Credits

hwloc is the evolution and merger of the libtopology
(http://runtime.bordeaux.inria.fr/libtopology/) project and the Portable
Linux Processor Affinity (PLPA) (http://www.open-mpi.org/projects/plpa/)
project. Because of functional and ideological overlap, these two code bases
and ideas were merged and released under the name "hwloc" as an Open MPI
sub-project.

libtopology was initially developed by the INRIA Runtime Team-Project
(http://runtime.bordeaux.inria.fr/) (headed by Raymond Namyst
(http://dept-info.labri.fr/~namyst/). PLPA was initially developed by
the Open MPI development team as a sub-project. Both are now deprecated
in favor of hwloc, which is distributed as an Open MPI sub-project.

\htmlonly
</div>
\endhtmlonly



\page termsanddefs Terms and Definitions 

<dl>

<dt>Object</dt>
  <dd>Interesting kind of part of the system, such as a Core, a Cache,
  a Memory node, etc. The different types detected by hwloc are
  detailed in the ::hwloc_obj_type_t enumeration.
  
  They are topologically sorted by CPU set into a tree.
  </dd>

<dt>CPU set</dt>
  <dd>The set of logical processors (or processing units) logically included in an object
  (if it makes sense).  They are always expressed using physical logical
  processor numbers (as announced by the OS).  They are just masks, they do \em
  not have any relation with an operating system actual binding notion like
  Linux' cpusets.</dd>

<dt>Parent object</dt>
  <dd>The object logically containing the current object, for example
  because its CPU set includes the CPU set of the current object.</dd>

<dt>Ancestor object</dt>
  <dd>The parent object, or its own parent object, and so on.</dd>

<dt>Children object(s)</dt>
  <dd>The object (or objects) contained in the current object because
  their CPU set is included in the CPU set of the current object.</dd>

<dt>Arity</dt>
  <dd>The number of children of an object.</dd>

<dt>Sibling objects</dt>
  <dd>Objects of the same type which have the same parent.</dd>

<dt>Sibling rank</dt>
  <dd>Index to uniquely identify objects of the same type which have
  the same parent, and is always in the range [0, parent_arity).</dd>

<dt>Cousin objects</dt>
  <dd>Objects of the same type as the current object.</dd>

<dt>Level</dt>
  <dd>Set of objects of the same type.</dd>

<dt>OS index</dt>
  <dd>The index that the operating system (OS) uses to identify the
  object.  This may be completely arbitrary, or it may depend on the
  BIOS configuration.</dd>

<dt>Depth</dt>
  <dd>Nesting level in the object tree, starting from the 0th object.</dd>

<dt>Logical index</dt>
  <dd>Index to uniquely identify objects of the same type.  It
  expresses proximity in a generic way.  This index is always linear
  and in the range [0, num_objs_same_type_same_level).  Think of it
  as ``cousin rank.'' The ordering is based on topology first, and
  then on OS CPU numbers, so it is stable across everything except
  firmware CPU renumbering.</dd>

<dt>Logical processor</dt>
<dt>Processing unit</dt>
  <dd>The smallest processing element that can be represented by a hwloc
  object. It may be a single-core processor, a core of a multicore
  processor, or a single thread in SMT processor.</dd>

</dl>

The following diagram can help to understand the vocabulary of the relationships
by showing the example of a machine with two dual core sockets (with no
hardware threads); thus, a topology with 4 levels. Each box with rounded corner
corresponds to one hwloc_obj_t, containing the values of the different integer
fields (depth, logical_index, etc.), and arrows show to which other hwloc_obj_t
pointers point to (first_child, parent, etc.)

\image html diagram.png
\image latex diagram.eps width=\textwidth

It should be noted that for PU objects, the logical index -- as
computed linearly by hwloc -- is not the same as the OS index.



\page tools Command-line tools 

hwloc comes with an extensive C programming interface and several
command line utilities. Each of them is fully documented in its own
manual page; the following is a summary of the available command line
tools.

\section cli_lstopo lstopo

lstopo (also known as hwloc-info and hwloc-ls) displays the
hierarchical topology map of the current system.  The output may be
graphic or textual, and can also be exported to numerous file
formats such as PDF, PNG, XML, and others.

Note that lstopo can read XML files and/or alternate chroot
filesystems and display topological maps representing those systems
(e.g., use lstopo to output an XML file on one system, and then use
lstopo to read in that XML file and display it on a different system).

\section cli_hwloc_bind hwloc-bind

hwloc-bind binds processes to specific hardware objects through a
flexible syntax.  A simple example is binding an executable to
specific cores (or sockets or cpusets or ...).  The hwloc-bind(1) man
page provides much more detail on what is possible.

hwloc-bind can also be used to retrieve the current process' binding.

\section cli_hwloc_calc hwloc-calc

hwloc-calc is generally used to create cpuset strings to pass to
hwloc-bind.  Although hwloc-bind accepts many forms of object
specification (i.e., cpuset strings are one of many forms that
hwloc-bind understands), they can be useful, compact representations
in shell scripts, for example.

hwloc-calc generates cpuset strings from given hardware objects with
the ability to aggregate them, intersect them, and more.  hwloc-calc
generally uses the same syntax than hwloc-bind, but multiple instances
may be composed to generate complex combinations.  

Note that hwloc-calc can also generate lists of logical processors or
NUMA nodes that are convenient to pass to some external tools such as
taskset or numactl.

\section cli_hwloc_distrib hwloc-distrib

hwloc-distrib generates a set of cpuset strings that are uniformly
distributed across the machine for the given number of processes.
These strings may be used with hwloc-bind to run processes to maximize
their memory bandwidth by properly distributing them across the
machine.



\page envvar Environment variables

The behavior of the hwloc library and tools may be tuned thanks to the
following environment variables.

<dl>

<dt>HWLOC_XMLFILE=/path/to/file.xml</dt>
  <dd>enforces the discovery from the given XML file as if
  hwloc_topology_set_xml() had been called.
  This file may have been generated earlier with lstopo file.xml.
  For convenience, this backend provides empty binding hooks which just
  return success.  To have hwloc still actually call OS-specific hooks,
  HWLOC_THISSYSTEM should be set 1 in the environment too, to assert that
  the loaded file is really the underlying system.
  </dd>

<dt>HWLOC_FSROOT=/path/to/linux/filesystem-root/</dt>
  <dd>switches to reading the topology from the specified
  Linux filesystem root instead of the main file-system root, as if
  hwloc_topology_set_fsroot() had been called.
  Not using the main file-system root causes hwloc_topology_is_thissystem()
  to return 0.
  For convenience, this backend provides empty binding hooks which just
  return success.  To have hwloc still actually call OS-specific hooks,
  HWLOC_THISSYSTEM should be set 1 in the environment too, to assert that
  the loaded file is really the underlying system.
  </dd>

<dt>HWLOC_THISSYSTEM=1</dt>
  <dd>enforces the return value of hwloc_topology_is_thissystem().
  It means that it makes hwloc assume that the selected backend provides the
  topology for the system on which we are running, even if it is not the
  OS-specific backend but the XML backend for instance.
  This means making the binding functions actually call the OS-specific
  system calls and really do binding, while the XML backend would otherwise
  provide empty hooks just returning success.
  This can be used for efficiency reasons to first detect the topology once,
  save it to an XML file, and quickly reload it later through the XML
  backend, but still having binding functions actually do bind.
  </dd>

</dl>



\page interoperability Interoperability with other software

Although hwloc offers its own portable interface, it still may have to
interoperate with specific or non-portable libraries that manipulate
similar kinds of objects.  hwloc therefore offers several specific
"helpers" to assist converting between those specific interfaces and
hwloc.

Some external libraries may be specific to a particular OS; others may
not always be available.  The hwloc core therefore generally does not
explicitly depend on these types of libraries.  However, when a custom
application uses or otherwise depends on such a library, it may
optionally include the corresponding hwloc helper to extend the hwloc
interface with dedicated helpers.

<dl>

<dt>Linux specific features</dt>
 <dd>
  hwloc/linux.h offers Linux-specific helpers that utilize some
  non-portable features of the Linux system, such as binding threads
  through their thread ID ("tid") or parsing kernel CPU mask files.
 </dd>

<dt>Linux libnuma</dt>
 <dd>
  hwloc/linux-libnuma.h provides conversion helpers between hwloc CPU
  sets and libnuma-specific types, such as nodemasks and bitmasks.  It
  helps you use libnuma memory-binding functions with hwloc CPU sets.
 </dd>

<dt>Glibc</dt>
 <dd>
  hwloc/glibc-sched.h offers conversion routines between Glibc and
  hwloc CPU sets in order to use hwloc with functions such as
  sched_setaffinity().
 </dd>

<dt>OpenFabrics Verbs</dt>
 <dd>
  hwloc/openfabrics-verbs.h helps interoperability with the
  OpenFabrics Verbs interface.  For example, it can return a list of
  processors near an OpenFabrics device.
 </dd>

</dl>



\page threadsafety Thread safety

Like most libraries that mainly fill data structures, hwloc is not
thread safe but rather reentrant: all state is held in a \ref
hwloc_topology_t instance without mutex protection.  That means, for
example, that two threads can safely operate on and modify two
different \ref hwloc_topology_t instances, but they should not
simultaneously invoke functions that modify the <em>same</em>
instance.  Similarly, one thread should not modify a \ref
hwloc_topology_t instance while another thread is reading or
traversing it.  However, two threads can safely read or traverse the
same \ref hwloc_topology_t instance concurrently.

When running in multiprocessor environments, be aware that proper thread
synchronization and/or memory coherency protection is needed to pass hwloc
data (such as \ref hwloc_topology_t pointers) from one processor
to another (e.g., a mutex, semaphore, or a memory barrier).
Note that this is not a hwloc-specific requirement, but it is worth
mentioning.

For reference, \ref hwloc_topology_t modification operations include
(but may not be limited to):

<dl>

<dt>Creation and destruction</dt>
  <dd><tt>hwloc_topology_init(), hwloc_topology_load(),
  hwloc_topology_destroy()</tt> (see \ref hwlocality_creation) imply
  major modifications of the structure, including freeing some
  objects.  No other thread cannot access the topology or any of its
  objects at the same time.

  Also references to objects inside the topology are not valid anymore
  after these functions return.  </dd>

<dt>Runtime topology modifications</dt>
  <dd><tt>hwloc_topology_insert_misc_object_by_*</tt> (see \ref
  hwlocality_tinker) may modify the topology significantly by adding
  objects inside the tree, changing the topology depth, etc.

  Although references to former objects <em>may</em> still be valid
  after insertion, it is strongly advised to not rely on any such
  guarantee and always re-consult the topology to reacquire new
  instances of objects.  </dd>

<dt>Locating topologies</dt>

  <dd><tt>hwloc_topology_ignore*</tt>, <tt>hwloc_topology_set*</tt>
  (see \ref hwlocality_configuration) do not modify the topology
  directly, but they do modify internal structures describing the
  behavior of the next invocation of <tt>hwloc_topology_load()</tt>.
  Hence, all of these functions should not be used concurrently.

  Note that these functions do not modify the current topology until
  it is actually reloaded; it is possible to use them while other
  threads are only read the current topology.  </dd>

</dl>



\page embed Embedding hwloc in other software

It can be desirable to include hwloc in a larger software package (be
sure to check out the LICENSE file) so that users don't have to
separately download and install it before installing your software.
This can be advantageous to ensure that your software uses a
known-tested/good version of hwloc, or for use on systems that do not
have hwloc pre-installed.

When used in "embedded" mode, hwloc will:

- not install any header files
- not build any documentation files
- not build or install any executables or tests
- not build <tt>libhwloc.*</tt> -- instead, it will build
  <tt>libhwloc_embedded.*</tt>

There are two ways to put hwloc into "embedded" mode.  The first is
directly from the configure command line:

\verbatim
shell$ ./configure --enable-embedded-mode ...
\endverbatim

The second requires that your software project uses the GNU Autoconf /
Automake / Libtool tool chain to build your software.  If you do this,
you can directly integrate hwloc's m4 configure macro into your
configure script.  You can then invoke hwloc's configuration tests and
build setup by calling an m4 macro (see below).

\section embedding_m4 Using hwloc's M4 Embedding Capabilities

Every project is different, and there are many different ways of
integrating hwloc into yours.  What follows is <em>one</em> example of
how to do it.

If your project uses recent versions Autoconf, Automake, and Libtool
to build, you can use hwloc's embedded m4 capabilities.  We have
tested the embedded m4 with projects that use Autoconf 2.65, Automake
1.11.1, and Libtool 2.2.6b.  Slightly earlier versions of may also
work but are untested.  Autoconf versions prior to 2.63 are almost
certain to not work because hwloc uses macros that were introduced in
2.63.

You can either copy all the config/hwloc*m4 files from the hwloc
source tree to the directory where your project's m4 files reside, or
you can tell aclocal to find more m4 files in the embedded hwloc's
"config" subdirectory (e.g., add "-Ipath/to/embedded/hwloc/config" to
your Makefile.am's ACLOCAL_AMFLAGS).

The following macros can then be used from your configure script (only
HWLOC_INIT <em>must</em> be invoked if using the m4 macros):

- HWLOC_INIT(config-dir-prefix, action-upon-success,
  action-upon-failure): Invoke the hwloc configuration tests and setup
  the hwloc tree to build.  The first argument is the prefix to use
  for AC_OUTPUT files -- it's where the hwloc tree is located relative
  to <tt>$top_srcdir</tt>.  Hence, if your embedded hwloc is located
  in the source tree at contrib/hwloc, you should pass
  <tt>[contrib/hwloc]</tt> as the first argument.  If HWLOC_INIT and
  the rest of <tt>configure</tt> completes successfully, then "make"
  traversals of the hwloc tree with standard Automake targets (all,
  clean, install, etc.) should behave as expected.  For example, it is
  safe to list the hwloc directory in the SUBDIRS of a higher-level
  Makefile.am.  <strong>NOTE: If the HWLOC_SET_SYMBOL_PREFIX macro is
  used, it must be invoked <em>before</em> HWLOC_INIT.</strong>

- HWLOC_SET_SYMBOL_PREFIX(foo_): Tells the hwloc to prefix all of
  hwloc's types and public symbols with "foo_"; meaning that function
  hwloc_init() becomes foo_hwloc_init().  Enum values are prefixed
  with an upper-case translation if the prefix supplied;
  HWLOC_OBJ_SYSTEM becomes FOO_HWLOC_OBJ_SYSTEM.  This is recommended
  behavior if you are including hwloc in middleware -- it is possible
  that your software will be combined with other software that links
  to another copy of hwloc.  If both uses of hwloc utilize different
  symbol prefixes, there will be no type/symbol clashes, and
  everything will compile, link, and run successfully.  If you both
  embed hwloc without changing the symbol prefix and also link against
  an external hwloc, you may get multiple symbol definitions when
  linking your final library or application.

- HWLOC_DO_AM_CONDITIONALS: If you embed hwloc in a larger project and
  build it conditionally (e.g., if HWLOC_INIT is invoked
  conditionally), you must unconditionally invoke
  HWLOC_DO_AM_CONDITIONALS to avoid warnings from Automake (for the
  cases where hwloc is not selected to be built).  This macro is
  necessary because hwloc uses some AM_CONDITIONALs to build itself,
  and AM_CONDITIONALs cannot be defined conditionally.  Note that it
  is safe (but unnecessary) to call HWLOC_DO_AM_CONDITIONALS even if
  HWLOC_INIT is invoked unconditionally.

NOTE: When using the HWLOC_INIT m4 macro, it may be necessary to
explicitly invoke AC_CANONICAL_TARGET and/or AC_USE_SYSTEM_EXTENSIONS
macros early in the configure script (e.g., after AC_INIT but before
AM_INIT_AUTOMAKE).  See the Autoconf documentation for further
information.


\section embedding_example Example Embedding hwloc

Here's an example of integrating with a larger project named sandbox
that already uses Autoconf, Automake, and Libtool to build itself:

\verbatim
# First, cd into the sandbox project source tree
shell$ cd sandbox
shell$ cp -r /somewhere/else/hwloc-<version> my-embedded-hwloc
shell$ edit Makefile.am
  1. Add "-Imy-embedded-hwloc/config" to ACLOCAL_AMFLAGS
  2. Add "my-embedded-hwloc" to SUBDIRS
  3. Add "$(HWLOC_EMBEDDED_LDADD)" and "$(HWLOC_EMBEDDED_LIBS)" to 
     sandbox's executable's LDADD line.  The former is the name of the 
     Libtool convenience library that hwloc will generate.  The latter 
     is any dependent support libraries that may be needed by 
     $(HWLOC_EMBEDDED_LDADD).
  4. Add "$(HWLOC_EMBEDDED_CPPFLAGS)" to AM_CPPFLAGS
shell$ edit configure.ac
  1. Add "HWLOC_SET_SYMBOL_PREFIX(sandbox_hwloc_)" line
  2. Add "HWLOC_INIT([my-embedded-hwloc], [happy=yes], [happy=no])" line
  3. Add error checking for happy=no case
shell$ edit sandbox.c
  1. Add #include <hwloc.h>
  2. Add calls to sandbox_hwloc_init() and other hwloc API functions
\endverbatim

Now you can bootstrap, configure, build, and run the sandbox as normal
-- all calls to "sandbox_hwloc_*" will use the embedded hwloc rather
than any system-provided copy of hwloc.



\page switchfromplpa Switching from PLPA to hwloc

Although PLPA and hwloc share some of the same ideas, their
programming interfaces are quite different.  After much debate, it was
decided <em>not</em> to emulate the PLPA API with hwloc's API because
hwloc's API is already far more rich than PLPA's.  

More specifically, exploiting modern computing architecture
<em>requires</em> the flexible functionality provided by the hwloc API
-- the PLPA API is too rigid in its definitions and practices to
handle the evolving server hardware landscape (e.g., PLPA only
understands cores and sockets; hwloc understands a much larger set
of hardware objects).

As such, even though it is fully possible to emulate the PLPA API with
hwloc (e.g., only deal with sockets and cores), and while the
documentation below describes how to do this, we encourage any
existing PLPA application authors to actually re-think their
application in terms of more than just sockets and cores.  In short,
we encourage you to use the full hwloc API to exploit <em>all</em> the
hardware.


\section switchfromplpa_caching Topology Context vs. Caching

First, all hwloc functions take a \p topology parameter. This
parameter serves as an internal storage for the result of the topology
discovery. It replaces PLPA's caching abilities and even lets you
manipulate multiple topologies as the same time, if needed.

Thus, all programs should first run hwloc_topology_init() and
hwloc_topology_destroy() as they did plpa_init() and plpa_finalize()
in the past.


\section switchfromplpa_hierarchy Hierarchy vs. Core@Socket

PLPA was designed to understand only cores and sockets.  hwloc offers
many more different types of objects (e.g., cores, sockets, hardware
threads, NUMA nodes, and others) and stores them within a tree of
resources.

To emulate the PLPA model, it is possible to find sockets using
functions such as hwloc_get_obj_by_type().  Iterating over sockets is
also possible using hwloc_get_next_obj_by_type().  Then, finding a
core within a socket may be done using
hwloc_get_obj_inside_cpuset_by_type() or
hwloc_get_next_obj_inside_cpuset_by_type().

It is also possible to directly find an object "below" another object
using hwloc_get_obj_below_by_type() (or
hwloc_get_obj_below_array_by_type()).


\section switchfromplpa_indexes Logical vs. Physical/OS Indexes

hwloc manipulates logical indexes, meaning indexes specified with
regard to the ordering of objects in the hwloc-provided hierarchical
tree.  Physical or OS indexes may be entirely hidden if not strictly
required.  The reason for this is that physical/OS indexes may change
with the OS or with the BIOS version.  They may be non-consecutive,
multiple objects may have the same physical/OS indexes, making their
manipulation tricky and highly non-portable.

Note that hwloc tries very hard to always present a hierarchical tree
with the same logical ordering, regardless of physical or OS index
ordering.

It is still possible to retrieve physical/OS indexes through the \p
os_index field of objects, but such practice should be avoided as much
as possible for the reasons described above (except perhaps for
prettyprinting / debugging purposes).

::HWLOC_OBJ_PU objects are supposed to have different physical/OS
indexes since the OS uses them for binding.  The \p os_index field of
these objects provides the identifier that may be used for such
binding, and hwloc_get_proc_obj_by_os_index() finds the object
associated with a specific OS index.  

But as mentioned above, we discourage the use of these conversion
methods for actual binding.  Instead, hwloc offers its own binding
model using the \p cpuset field of objects.  These cpusets may be
duplicated, modified, combined, etc. (see hwloc/cpuset.h for details)
and then passed to hwloc_set_cpubind() for binding.


\section switchfromplpa_counting Counting Specification

PLPA offers a countspec parameter to specify whether counting all
CPUs, only the online ones or only the offline ones.  However, some
operating systems do not expose the topology of offline CPUs (i.e.,
offline CPUs are not reported at all by the OS).  Also, some
processors may not be visible to the current application due to
administrative restrictions.  Finally, some processors let you
shutdown a single hardware thread in a core, making some of the PLPA
features irrelevant.

hwloc stores in the hierarchical tree of objects all CPUs that have
known topology information.  It then provides the applications with
several cpusets that contain the list of CPUs that are actually known,
that have topology information, that are online, or that are available
to the application.  These cpusets may be retrieved with
hwloc_topology_get_online_cpuset() and other similar functions to
filter the object that are relevant or not.

*/
